Increased resolution and dynamic range capture unit in a surgical instrument and method

ABSTRACT

In a minimally invasive surgical system, an image capture unit includes a prism assembly and sensor assembly. The prism assembly includes a beam splitter, while the sensor assembly includes coplanar image capture sensors. Each of the coplanar image capture sensors has a common front end optical structure, e.g., the optical structure distal to the image capture unit is the same for each of the sensors. A controller enhances images acquired by the coplanar image capture sensors. The enhanced images may include (a) visible images with enhanced feature definition, in which a particular feature in the scene is emphasized to the operator of minimally invasive surgical system; (b) images having increased image apparent resolution; (c) images having increased dynamic range; (d) images displayed in a way based on a pixel color component vector having three or more color components; and (e) images having extended depth of field.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/139,288 (filed 24 Sep. 201.8), which is a continuation of U.S. patentapplication Ser. No. 14/258,275 (filed 22 Apr. 2014; now U.S. Pat. No.10,254,533), which is a divisional of U.S. patent application Ser. No.13/209,084 (filed 12 Aug. 2011; now U.S. Pat. No. 8,734,328 B2), each ofwhich is incorporated herein by reference.

BACKGROUND Field of Invention

Aspects of this invention are related generally to endoscopic imagingand are more particularly related to capturing light from a common frontend optical structure in a plurality of coplanar image capture sensors.

Related Art

The da Vinci® Surgical Systems, commercialized by Intuitive Surgical,Inc., Sunnyvale, Calif., are minimally invasive teleoperated surgicalsystems that offer patients many benefits, such as reduced trauma to thebody, faster recovery and shorter hospital stay. One key component of ada Vinci® Surgical System (e.g., the model IS3000, da Vinci® Si HD) is acapability to provide two-channel (i.e., left and right) video captureand display of visible images to provide stereoscopic viewing for thesurgeon. Such electronic stereoscopic imaging systems may output highdefinition video images to the surgeon, and may allow features such aszoom to provide a “magnified” view that allows the surgeon to identifyspecific tissue types and characteristics, as well as to work withincreased precision.

Typically in a minimally invasive surgical system, an image capturesystem is coupled to a proximal end (away from the surgical site) of astereoscopic endoscope. However, some stereoscopic endoscopes haveincluded image capture components in the distal. end (nearest thesurgical site) of the endoscope. FIGS. 1A to 1D are examples of imagecapture sensor configurations in a distal end of a stereoscopicendoscope from U.S. Pat. No. 4,873,572 (filed Feb. 24, 1988).

In FIG. 1A, a distal end 100A of an endoscope includes a plate-likepackage 113A with a center line coinciding with a longitudinal axis 133Aof the endoscope. Two charge coupled devices (CCDs) 114A1 and 114A2 aremounted on opposing surfaces of package 113A. Two objective lenses 115A1and 115A2 are symmetrically arranged on both sides of longitudinal axis133A of the endoscope. Mirrors 116A1., 116A2 are symmetrically arrangedon the optical axis of the respective objective lenses 115A1, 115A2.Light reflected from an object external to the endoscope passes throughobjective lens 115A1, 115A2 and is reflected by mirrors 116A1, 116A2onto the imaging surfaces of CCDs 114A1 and 114A2, The video signalsfrom CCDs 114A1 and 114A2 are transmitted to a video processor externalto the endoscope.

In FIG. 1B, a distal end 100B of an endoscope includes two objectivelenses 115B1, 115B2 arranged the same as objective lenses 115A1 and115A2 in FIG. 1A. Mirrors 116B1 and 116B2 are mounted with the mirrorsurfaces parallel to and removed from longitudinal axis 133BB of theendoscope. Light reflected from an object external to the endoscopepasses through objective lenses 115B1, 115B2 and is reflected by mirrors116B1, 116B2 to refracting prisms 117B1, 117B2. The optical path fromprisms 117B1, 117B2 is to the imaging surfaces of CCDs 114B1, 114B2,CCDs 114B1. and 114B2 are mounted so that the imaging surfaces of CCDs114B1, 114B2 intersect at right angles with the optical axis of theoptical path from prisms 117B1, 117B2, respectively. Thus, CCD 114B1 and114B2 are each mounted with the imaging surface inclined at apredetermined angle with respect to longitudinal axis 13313 of theendoscope.

In FIG. 1C, two objective lenses 115C1 and 115C2 are eccentric, forexample, to the upper side from the center axis of the lens. Reflectingprisms 117C1 and 117C2 are arranged on the optical axes of therespective objective lenses 115C1 and 115C2. The centers of Prisms 115C1and 115C2 are positioned at a same height as the respective objectivelenses 115C1 and 115C2, but are somewhat displaced in the horizontaldirection. Prism 117C1. is somewhat displaced to the left from objectivelens 115C1 and the prism 117C2 is somewhat displaced to the right fromobjective lens 115C2.

The light reflected by each of prisms 117c1 and 117C2 is reflected bythe respective slopes of prism 118C to form an image on the imagingsurface of CCD 114C fitted to package 113C. Video signals from CCD 114Care transmitted to a video processor external to the endoscope.

In FIG. 1D, a distal end 100D of an endoscope includes two eccentricobjective lenses 115D1, 115d2 arranged the same as objective lenses115C1 and 115C2 in FIG. 1C. The positions of prisms 117D1 and 117D2 aredisplaced forward and rearward in comparison to prisms 117C1 and 117C2in FIG. 1C. The light from prisms 117D1 and 117D2 is reflectedrespectively by mirrors 118D1 and 118D2 to form respective images onCCDs 114D1 and 114D2 mounted adjacently on package 113D, which isparallel to longitudinal axis of the endoscope.

One mirror 118D1 is concave and so forms an image on CCD 114D1 for asomewhat shorter optical path length than the optical path length forthe image on CCD 114D2. Hence, in this example, the left optical channelhas a shorter optical path length than the right optical channel. Videosignals from CCDs 114D1 and 114D2 are transmitted to a video processorexternal to the endoscope.

FIGS. 1A to 1D illustrate a few ways of capturing a stereo image in theconstrained space of an endoscope tip. But since a small outer diameterof the endoscope distal end is desirable, the configurations in thesefigures also illustrate how difficult it is to capture high qualitystereoscopic images in small outer diameter distal-end image capturesystems due to many problems.

Consider the configuration in FIG. 1A. To focus this device one has tomove the tiny lenses of both objective lenses 115A1 and 115A2 veryprecisely to obtain focus. The configuration in FIG. 1B suffers fromneeding to bend the light at an odd angle with a prism. This likelyleads to lateral color distortion and uneven performance on the left andright image sensors. The images are not optimally spaced.

The configurations in FIGS. 1C and 1D require the image to lie flat inthe plane of the optics. Either the CCD or the optical components cannotlie on the mid-plane of a round endoscope tip thus these configurationsrequire either very small optical components (and a smallinter-pupillary distance) or a very small CCD which limits the imagingquality as the area is small thus restricting the number of pixelsand/or the pixel size. Also, in the configuration of FIG. 1D, theoptical path lengths have different lengths, and so the opticalcomponents for each channel must be different.

SUMMARY

An image capture unit with coplanar image capture sensors overcomesshortcomings of the prior art cameras used in the distal end of anendoscope and provides many new capabilities. Each of the coplanar imagecapture sensors in a channel of the endoscope has a common front endoptical structure, e.g., the lens assembly in the image capture unit isthe same for each of the sensors. The common optical and coplanarconfiguration of the image capture sensors eliminates the need forcalibration of lens artifacts. Re-registration of different imagescaptured in independent channels of the endoscope is not required. Theimages captured in a channel of the endoscope are temporally registered.The images are also spatially registered relative to each other.

Visible images of a scene or a visible image of a scene and one or morefluorescence images in the scene are acquired by the image capture unit.A controller enhances the acquired images. The enhanced images aredisplayed on a stereoscopic display, in one aspect. The enhanced imagesmay include (a) visible images with enhanced feature definition, inwhich a particular feature in the scene is emphasized to the operator ofa minimally invasive surgical system, for example; (b) images havingincreased image apparent resolution; (c) images having increased dynamicrange; (d) images displayed in a way based on a pixel color componentvector having three or more color components; and (e) images havingextended depth of field.

In one aspect, an image capture unit includes a first image capturesensor with a first sensor surface and a second image capture sensorwith a second sensor surface. The first and second sensor surfaces arecoplanar. In another aspect, the first surface is in a first plane andthe second surface is in a second plane. The first and second planes areparallel, and are separated by a known distance. A beam splitter, in theimage capture unit, is positioned to receive light. The beam splitterdirects a first portion of the received light to the first sensorsurface and passes a second portion of the received light through thebeam splitter. A reflective unit, in the image capture unit, ispositioned to receive the second portion of the received light and todirect the second portion of the received light to the second imagecapture sensor.

In one aspect, the first and second image capture sensors are differentareas on an image capture sensor chip. In another aspect, the first andsecond image capture sensors are two separate image capture sensor chipsmounted on a common platform. In yet another aspect, the first andsecond image capture sensors are two separate imaging areas on a singleimage capture sensor chip.

In one aspect, a distal end of an endoscope includes the first andsecond image capture sensors, a prism assembly including the beamsplitter, and the reflective unit. In another aspect, a stereoscopicendoscope includes a distal end, a pair of channels, and a plurality offirst and second image capture sensors, prism assemblies, and reflectiveassemblies. The first image capture sensor, the second image capturesensor, the prism assembly, and the reflective unit are included in theplurality. Each channel in the pair of channels includes, in the distalend of the stereoscopic endoscope, a different first image capturesensor, a different second image capture sensor, a different prismassembly, and a different reflective unit in the plurality.

In one implementation, the beam splitter is included in a prism assemblythat also includes a surface positioned to direct the first portion oflight received from the beam splitter onto the first sensor surface.This surface is positioned so that no other light hits the surface. Thereflective unit includes a reflective surface positioned to reflect thesecond portion of the received light onto the surface of the secondimage capture sensor. In another implementation, the prism assembly andthe reflective unit are included in a single integral structure.

In one aspect, the prism assembly includes a distal face through whichthe received light enters the prism assembly. The image capture unit hasa first optical path length from the distal face to the first sensorsurface that is about equal to a second optical path length from thedistal face to the second sensor surface. In another aspect, the firstand second optical path lengths have different lengths and thedifference in length of the two optical path lengths is configured toprovide a difference in focus between the images acquired by the firstimage capture sensor and the second image capture sensor.

Independent of the implementation, the prism assembly includes a beamsplitter configured to reflect the first portion of the light receivedby the prism assembly and to transmit the second portion of the receivedlight. In one aspect, the first portion of the received light is a firstpercentage of the received light, and the second portion of the receivedlight is a second percentage of the received light. In one aspect, thebeam splitter is configured so that the first and second percentages areabout equal. In another aspect, the beam splitter is configured so thatthe first and second percentages are not equal. The beam splitter may beimplemented in many ways, including but not limited to, thin metalliccoatings, dielectric coatings, dichroic coatings, or a pattern ofreflective tiles on an otherwise transparent interface.

The first and second image capture sensors can both be color imagecapture sensors or alternatively, one of the image sensors is a colorimage sensor and the other of the image captures sensors is a monochromeimage capture sensor.

With the image capture unit, a first image from a first portion of lightreceived from a common front end optical system is captured by the firstimage capture sensor. A second image from a second portion of the lightreceived from the common front end optical system is captured by thesecond image capture sensor. The first and second image capture sensorsare coplanar and the first and second images are registered spatiallyrelative to each other upon being captured.

The same basic geometry for the prism assembly and the reflective unit,is used in each of the various aspects to achieve the advantagesdescribed above. Depending on the particular enhancement, theconfiguration of the beam splitter is varied and the illumination sourcemay be varied.

For enhanced feature differentiation, the light received from the lensassembly enters the prism assembly through a distal face. The beamsplitter is configured to reflect a first portion of the received lighton a basis of a polarization state of the received light, and totransmit a second portion of the received light on the basis of thepolarization state of the received light. A first optical path lengthfrom the distal face to the first sensor surface is about equal to asecond optical path length from the distal face to the second sensorsurface. A controller is coupled to the first and second image capturesensors. The controller combines information from a first image capturedby the first image capture sensor and information from. a second imagecaptured by the second image capture sensor to generate an imageincreasing the saliency of a feature in the image based on polarizationdifferences in the received light.

For enhanced resolution and dynamic ranges, the beam splitter isconfigured to reflect a first percentage of the received light, and totransmit a second percentage of the received light. Again, a firstoptical path length from a distal face of the prism assembly to thefirst sensor surface is about equal to a second optical path length fromthe distal face to the second sensor surface.

In one aspect, the first and second image capture sensors are colorimage capture sensors. Again, the controller is coupled to the first andsecond image capture sensors. The controller combines information from afirst image captured by the first image capture sensor and informationfrom a second image captured by the second image capture sensor togenerate an image having one of enhanced spatial resolution and enhanceddynamic range relative to an image captured by a single image capturesensor.

When the first and second percentages are about equal, the controllergenerated image has enhanced spatial resolution. When the first andsecond percentages are not about equal, and the controller generatedimage has enhanced dynamic range.

For the enhanced resolution in one aspect, the first percentage is aboutfifty percent of the received light and the second percentage is aboutfifty percent of the received light. The beam splitter and reflectivesurface of the prism assembly are positioned to offset an image capturedby the first image capture sensor from an image captured by the secondimage capture sensor with the first optical path length remaining aboutequal to the second optical path length. The controller samples a firstpixel in a first image captured by the first image capture sensor andsamples a second pixel captured by the second image capture sensor andcorresponding to the first pixel. Using information from the two sampledpixels, the controller generates a pixel in an image having increasedcolor performance in comparison to the images captured by the first andsecond image capture sensors. The controller may perform this processusing groups of pixels instead of single pixels.

When the beam splitter separates the received light so that the firstpercentage and the second percentage are not equal, the first percentageis selected based on the dynamic range of the first image capturesensor, e.g., so that an image captured by the first image capturesensor is not clipped due to the dynamic range of the first imagecapture sensor. In one aspect, the first percentage is about N % of thereceived light and the second percentage is about M % of the receivedlight. N and M are positive numbers. One hundred percent minus N % isabout equal to M %. The controller samples a pixel in an image capturedby the first image capture sensor and samples a corresponding pixel inan image captured by the second image capture sensor. The controlleruses the information from the sampled pixels to generate a pixel in anoutput image. The output image has an increased dynamic range relativeto an image captured by a single image capture sensor. The controllermay perform this process using groups of pixels instead of singlepixels.

In another aspect of enhanced resolution, the light received by the beamsplitter includes a plurality of color components. The beam splitter isconfigured to reflect one color component of the plurality of colorcomponents and to transmit other color components in the plurality ofcolor components. The first optical path length from the distal face ofthe prism assembly to the first sensor surface is about equal to thesecond optical path length from the distal face to the second sensorsurface.

In this aspect, the first image capture sensor is a monochrome imagecapture sensor, and the second image capture sensor is an image capturesensor having a color filter array for the other color components in theplurality of color components. The controller has full spatialresolution in the one of the plurality of color components and hasreduced spatial resolution in the other of the plurality of colorcomponents. The controller generates an image having improved spatialresolution and sharpness relative to an image captured by a color imagecapture sensor.

For aspects including a pixel color component vector having three ormore color components, the beam splitter includes a plurality of notchfilters. A notch filter is a filter with a spectrally narrow band inwhich the filter is reflective, and a broader pass band, which may be onone side or both sides of the reflective band. The plurality of notchfilters reflects a first set of light components as a first, portion ofthe received light and passes a second set of light components as asecond portion of the received light. Again, the first optical pathlength from the distal face of the prism assembly to the first sensorsurface is about equal to a second optical path length from the distalface to the second sensor surface.

The system includes an illuminator that generates output light includinga plurality of color components. The controller is configured to receivea demosaiced image of a first image captured by the first image capturesensor, and configured to receive a demosaiced image of a second imagecaptured by the second image capture sensor. The controller generates anN-element color component vector for a pixel in an output image from acolor component vector of a corresponding pixel in the first dernosaicedimage and a color component vector of a corresponding pixel in thedemosaiced second image, where N is at least three.

For the extended depth of field aspect, the beam splitter reflects afirst portion of the received light and transmits a second portion ofthe received light. A first optical path length from a distal face ofthe prism assembly to the first sensor surface is smaller than a secondoptical path length from the distal face to the second sensor surface.The first image capture sensor captures an image focused at a firstobject distance, and the second image capture sensor captures an imagefocused at a second object distance. In one aspect, the controller iscoupled to the first and second image capture sensors to receive thefirst and second images. The controller automatically shifts an outputimage between the first image and the second image as the object toendoscope distance changes without physically moving the optics insidethe endoscope. In another aspect, the controller is configured to samplea region of pixels in the first image and to sample a correspondingregion of pixels in the second image to generate pixels in an outputimage having increased apparent depth of field in comparison to thefirst and second images individually. In still yet another aspect, thecontroller combines a first image captured by the first image capturesensor with a second image captured by the second image capture sensor,and generates a third image that automatically stays in focus duringphysical motion of the endoscope relative to the tissue being viewed.This is accomplished by the controller processing regions of the firstand second images and comparing their sharpness. The controller createsthe third image from the pixels in the sharper of the two images at eachof the regions. The third image is thus constructed from the sharpestportions of the two images.

In a further aspect, the controller retrieves a first image captured bythe first image capture sensor and a second image captured by the secondimage capture sensor, and generates a channel depth map based on therelative sharpness of pixel regions acquired from the first and secondimage capture sensors. The depth map may be used by the system invarious ways. One way is for the controller to generate athree-dimensional surface of a scene and then to project (by executingsoftware) and texture map the first and second images back on thethree-dimensional surface to generate a textured virtual image surface.The controller generates a new virtual image for a virtual camera pointfrom the channel depth map and the textured image surface. More than onevirtual camera position and corresponding image may be created ifdesired. For example, real-time images of a scene are generated as thevirtual camera position is swept back and forth from a left eye positionto a right eye position, i.e., swept back and forth across aninterocular separation. When an image from a virtual camera view pointis generated, the image is displayed on a non-stereo display unit. Theview point is moved to the next virtual camera position, and an imagefrom that view point is generated and displayed. Thus, as the virtualcamera position is swept back and forth, the displayed scene rocks backand forth over time and so gives depth cues to a person viewing thedisplay without requiring a stereoscopic viewer.

In yet another aspect, an apparatus includes a first image capturesensor and a second image capture sensor. The first capture image sensorhas a first sensor surface, while the second image capture sensor has asecond sensor surface. The apparatus also includes a first lens assemblyand a second lens assembly. A reflective unit is positioned to receivelight that passes through the first lens assembly and is positioned toreceive light that passes through the second lens assembly. Thereflective unit reflects the received light from the first lens assemblyunto the first sensor surface. The reflective unit also reflects thereceived light from the second lens assembly unto the second sensorsurface. A first optical path length from the first lens assembly to thefirst sensor surface is about equal to a second optical path length fromthe second lens assembly to the second sensor surface.

In one aspect, the first and second image capture sensor surfaces ofthis apparatus are coplanar. In another aspect, the first sensor surfaceis in a first plane. The second sensor surface is in a second plane, andthe first and second planes are substantially parallel and are separatedby a known distance. In both cases the first optical path length and thesecond optical path length are about equal.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A to 1D are prior art examples of image capture sensorconfigurations in a distal end of a stereoscopic endoscope.

FIG. 2 is a block diagram of a minimally invasive surgical system thatincludes a plurality of image capture units in a distal end of astereoscopic endoscope.

FIG. 3A is a block diagram of a distal end of a stereoscopic endoscopethat includes a plurality of image capture units.

FIG. 3B is a block diagram of a distal end of a monoscopic endoscopethat includes an image capture unit.

FIG. 4A is an illustration of a portion of an image capture unit thatincludes a lens assembly and a sensor assembly having coplanar imagecapture sensors.

FIG. 4B is an example of an implementation of the structure in FIG. 4Ausing a pentaprism as the prism assembly.

FIG. 5A is a schematic illustration of a distal end of a stereoscopicendoscope including an illumination channel, which provides unpolarizedlight from an illuminator, and left and right stereoscopic opticalchannels that each includes an image capture unit having a lens assemblyand a sensor assembly.

FIG. 5B is a schematic illustration of a distal end of a stereoscopicendoscope including an illumination channel that provides unpolarizedlight from an illuminator, a polarizer that polarizes the illuminationfrom the illumination channel, and left and right stereoscopic opticalchannels that each includes an image capture unit having a lens assemblyand a sensor assembly.

FIG. 6A is a schematic illustration of a distal end of a stereoscopicendoscope, an illumination channel that provides light from anilluminator, and left and right stereoscopic optical channels that eachincludes an image capture unit having a lens assembly and a sensorassembly. Each sensor assembly includes a beam splitter with a coatedsurface that reflects a first percentage of the received light and thatpasses a second percentage of the received light through the coatedsurface.

FIG. 6B is a schematic illustration of the offset for a block of pixelsin the first image and a corresponding block of pixels in the secondimage.

FIG. 7A is a schematic illustration of a distal end of a stereoscopicendoscope including an illumination channel that provides light from oneof a plurality of illuminators, and left and right stereoscopic opticalchannels that each includes an image capture unit having a lens assemblyand a sensor assembly.

FIGS. 7B to 7E are block illustrations of different illuminators thatcan be coupled to the stereoscopic endoscope of FIG. 7A as well as toany of the other endoscopes and devices described herein.

FIG. 7F is a graphical representation of a filter that includes aplurality of notch filters.

FIG. 7G is a graphical representation of a spectrum of light that isreceived by an image capture unit.

FIG. 7H is a graphical representation of another filter that includes aplurality of notch filters.

FIG. 8A is a schematic illustration of a distal end of a stereoscopicendoscope including an illumination channel that provides light from anilluminator, and left and right stereoscopic optical channels that eachincludes an image capture unit having a lens assembly and a sensorassembly. Each sensor assembly has two different optical path lengths.

FIGS. 8B and 8C are illustrations of two images captured in a sensorassembly different depths of field and different focus.

FIG. 8D is an illustration of one combination of the two images of FIGS.8B and 8C combined to form an in focus image with extended depth offield.

FIG. 8E shows the sharpness of the images acquired by the first andsecond image capture sensors versus object distance.

FIG. 8F is a process flow diagram of a method to generate a virtualimage at a virtual camera view point from images captured by an imagecapture unit.

FIG. 9 is an illustration of a portion of an image capture unit thatincludes a lens assembly and a sensor assembly having image capturesensors that are not coplanar.

FIG. 10 is a schematic illustration of a distal end of a stereoscopicendoscope with a surface that directs light received from two lensassemblies to coplanar image sensors.

In the drawings, the first digit of a reference number indicates thefigure in which the element with that reference number first appearedfor single digit figure numbers. The first two digits of a referencenumber indicates the figure in which the element with that referencenumber first appeared for double digit figure numbers.

DETAILED DESCRIPTION

As used herein, electronic stereoscopic imaging includes the use of twoimaging channels (i.e., one channel for left side images and anotherchannel for right side images).

As used herein, a stereoscopic optical path includes two channels (e.g.,channels for left and right images) for transporting light from anobject such as tissue to be imaged. The light transported in eachchannel represents a different view (stereoscopic left or right) of ascene in the surgical field. Each one of the stereoscopic channels mayinclude one, two, or more optical. paths, and so light transported alonga single stereoscopic channel can form one or more images. For example,for the left stereoscopic channel, one left side image may be capturedfrom light traveling along a first optical path, and a second left sideimage may be captured from light traveling along a second optical path.Without loss of generality or applicability, the aspects described morecompletely below also could be used in the context of a field sequentialstereo acquisition system and/or a field sequential display system.

As used herein, an illumination channel includes a path providingillumination to tissue from an illumination source located away from animage capture unit (e.g., away from the distal end of an endoscope), oran illumination source located at or near the image capture unit (e.g.,one or more light emitting diodes (LEDs) at or near the distal end of anendoscope).

As used herein, white light is visible white light that is made up ofthree (or more) visible color components, e.g., a red visible colorcomponent, a green visible color component, and a blue visible colorcomponent. If the visible color components are provided by anilluminator, the visible color components are referred to as visiblecolor illumination components. White light may also refer to a morecontinuous spectrum in the visible spectrum as one might see from aheated tungsten filament or xenon lamp, for example.

As used herein, a visible image includes a visible color component.

As used herein, a non-visible image is an image that does not includeany of the visible color components. Thus, a non-visible image is animage formed by light outside the range typically considered visible.

As used herein, images captured as the result of fluorescence arereferred to as acquired fluorescence images. There are variousfluorescence imaging modalities. Fluorescence may result from naturaltissue fluorescence, or the use of, for example, injectable dyes,fluorescent proteins, or fluorescent tagged antibodies. Fluorescence mayresult from, for example, excitation by laser or other energy source. Insuch configurations, it is understood that a notch filter is used toblock the excitation wavelength that enters the endoscope. Fluorescenceimages can provide vital in vivo patient information that is criticalfor surgery, such as pathology information (e.g., fluorescing tumors) oranatomic information (e.g., fluorescing tagged tendons).

As used herein, the angle of incidence is the angle between a light rayincident on a surface and the line perpendicular to the surface at thepoint of incidence.

As used herein, images are processed digitally and may be re-oriented ormirrored by changing the way in which the image is indexed.Re-orientation or mirroring may also be accomplished in the order inwhich the image sensor is read.

Aspects of this invention facilitate acquiring visible and non-visiblestereoscopic images of a scene in a surgical field. Referring to FIG. 2,for example, image capture units 225L, 225R (FIG. 2) are located at adistal end of a stereoscopic endoscope 202 in a minimally invasivesurgical system 200, e.g., a da Vinci® minimally invasive teleoperatedsurgical system commercialized by Intuitive Surgical, Inc. of Sunnyvale,California. As indicated by arrow 235, the distal direction is towardstissue 203 and the proximal direction is away from tissue 203.

One image capture unit 225L captures left side images for a stereoscopicimage, sometimes referred to as left side stereoscopic images. A secondimage capture unit 225R captures right side images for a stereoscopicimage, sometimes referred to as right side stereoscopic images.

As described more completely below, each image capture unit includes alens assembly and a sensor assembly. The lens assembly is sometimesreferred to as a front end optical system. The sensor assembly includesa pair of coplanar image capture sensors, in one aspect, a foldedoptical path that transmits light from the lens assembly to one of thecoplanar image capture sensors, and another folded optical path thattransmits light from the lens assembly to the other coplanar imagecapture sensor. The same lens assembly in the image capture unit is usedfor both image capture sensors so that the image capture sensors aresaid to have a common front end optical structure. The combination ofthe shared lens assembly and the coplanar configuration of the imagecapture sensors eliminates the need for calibration to compensate forlens artifacts. Since the spatial relation between the two image capturesensors is constant, and since the image captures sensors share a commonlens assembly, spatial registration of a pair of images captured by thetwo image capture sensors remains constant over time and during changingoptical conditions, such as changing focus. A pair of images captured ina channel of endoscope 202 may also be temporally registered to eachother.

In one aspect, image capture units 225L, 225R are used in a minimallyinvasive surgical system that includes multiple viewing modes: a normalmode, and one or more enhanced modes. A person switches between theviewing modes by using display mode switch 252 that typically is in auser interface 262 presented on a surgeon's control console 250,sometimes referred to as surgeon's console 250.

In the normal viewing mode, visible images of a scene in the surgicalfield are acquired by image capture units 225L, 225R and displayed instereoscopic display 251 of a surgeon's control console 250. In anenhanced viewing mode, visible images of the scene or a visible image ofthe scene and one or more fluorescence images in the scene are acquiredby image capture units 225L, 225R, and dual image enhancement modules240R, 240L in a central controller 260 enhance the acquired images. Theenhanced images are displayed in stereoscopic display 251. The enhancedimages may include (a) visible images with enhanced feature definition,in which a particular feature in the scene is emphasized to the operatorof minimally invasive surgical system 200; (b) images having increasedimage apparent resolution; (c) images having increased dynamic range;(d) images displayed in a way based on a pixel color component vectorhaving three or more color components; and (e) images having extendeddepth of field.

Prior to considering image capture units 225L, 225R and the enhancedmodes of operation in further detail, minimally invasive surgical system200 is described. System 200 is illustrative only and is not intended tolimit the application of image capture units 225L, 225R to this specificsystem. Image capture units 225L, 225R could be implemented in variousother devices such as stereoscopic microscopes, monoscopic endoscopes,microscopes, and also could be used for replacement of existingendoscopic cameras.

Minimally invasive surgical system 200, for example, a da Vinci®Surgical System, includes image capture units 225L, 225R. In thisexample, a surgeon at surgeon's console 250 remotely manipulatesendoscope 202 that is mounted on a robotic manipulator arm (not shown).There are other parts, cables, etc. associated with the da Vinci®Surgical System, but these are not illustrated in FIG. 2 to avoiddetracting from the disclosure. Further information regarding minimallyinvasive surgical systems may be found for example in U.S. patentapplication Ser. No. 11/762,165 (filed Jun. 23, 2007; disclosing:Minimally invasive Surgical System), U.S. Pat. No. 6,837,883 B2 (filedOct. 5, 2001; disclosing Arm Cart for Telerobotic Surgical System), andU.S. Pat. No. 6,331,181 (filed Dec. 28, 2001; disclosing SurgicalRobotic Tools, Data Architecture, and Use), all of which areincorporated herein by reference.

An illuminator 210 is coupled to stereoscopic endoscope 202. Illuminator210 includes at least a white light source and optionally may includeone or more fluorescence excitation sources. Illuminator 210 is used inconjunction with at least one illumination channel in stereoscopicendoscope 202 to illuminate tissue 203. Alternatively and without lossof generality, illuminator 210 may be replaced by an illumination sourceat the distal tip, or near the distal tip, of endoscope 202. Such distaltip illumination may be provided by LEDs, for example, or otherillumination sources.

In one example, illuminator 210 provides white light illumination thatilluminates tissue 203 in white light. In some implementations,illuminator 210 can also provide non-visible light that excitesfluorescence and as well as a subset of the visible color componentsthat make-up white light.

Typically, three (or more) visible color components make up white light,e.g., white light includes a first visible color component, a secondvisible color component, and a third visible color component. Each ofthe three visible color components is a different visible colorcomponent, e.g., a red color component, a green color component, and ablue color component. Additional color components may also be used suchas cyan to improve color fidelity of the system.

In some implementations, a fluorescence excitation source in illuminator210 provides a fluorescence excitation illumination component thatexcites fluorescence in tissue 203. For example, narrow band light fromthe fluorescence excitation source is used to excite tissue-specificnear infrared emitting fluorophores so that fluorescence images ofspecific features within tissue 203 are acquired by image capture units225L, 225R.

Light from illuminator 210 is directed onto an illumination channel 226that couples illuminator 210 to the illumination channel in endoscope202. The illumination channel in stereoscopic endoscope 202 directs thelight to tissue 203. In another, aspect, an illumination source, such asLEDs or other sources, is provided at, or near the distal tip onendoscope 202. The illumination channels can be implemented with a fiberoptic bundle, a single stiff or flexible rod, or an optical fiber.

Each one of image capture units 225R, 225L in endoscope 202 include, inone aspect, a single lens assembly for passing light received fromtissue 203 to a sensor assembly. Light from tissue 203 may includevisible spectrum light components reflected from a white lightillumination source and fluorescence light (visible or non-visible) thatoriginates at tissue 203, for example as the result of receiving energyfrom a fluorescence excitation illumination source. The reflected whitelight components are used to capture an image or images that a viewerwould expect to see in the normal visible light spectrum.

Image capture unit 225L is coupled to a stereoscopic display 251 insurgeon's console 250 via a left camera control unit (CCU) 230L. Imagecapture unit 225R is coupled to stereoscopic display 251 in surgeon'sconsole 250 via a right camera control unit (CCU) 230R. Camera controlunits 230L, 230R receive signals from a system process module 263 thatcontrols gains, controls capturing images, controls transferringcaptures images to dual image enhancement modules 240R, 240L, etc.System process module 263 represents the various controllers includingthe vision system controllers in system 200. Camera control units 2301230R may be separate units, or may be combined in a single dualcontroller unit.

Display mode select switch 252 provides a signal to a user interface 262that in turn passes the selected display mode to system process module263. Various vision system controllers within system process module 263configure illuminator 210 to produce the desired illumination, configureleft and right camera. control units 230L and 230R to acquire thedesired images, and configure any other elements needed to process theacquired images so that the surgeon is presented the requested images indisplay 251.

Color correction modules 241L and 241R are in some embodiments each partof dual image enhancement modules 240L and 240R, respectively (describedin more detail below). Color correction modules 241L and 240L transformthe color of the acquired images to a new desired color balance asdetermined by system process module 263. As shown in FIG. 2, userinterface 262, system process module 263, and image enhancement modules240L, 240R are grouped as a central controller 260 for descriptivepurposes. Optional image processing module 264 receives video fromcentral controller 260 and processes images from color correctionmodules 241L and 241R prior to display on stereoscopic display 251 insurgeons console 250. Optional image processing module 264 is equivalentto image processing modules in prior art minimally invasive surgicalsystems and so is not considered in further detail.

FIG. 3A is a block diagram of a distal end of stereoscopic endoscope302A that includes image capture units 325L and 325R and an illuminationchannel 305, Each image capture unit 325R, 325L includes a lens assembly301R, 301L and a sensor assembly 320R, 320L. Sensor assembly 320R, 320Lis positioned to receive light that passes through lens assembly 301R,301L. Each image capture unit 320R, 320L includes a prism assembly 330R,330L, a reflective assembly 340R, 340L and coplanar image capturesensors (310R, 315R), (310L, 315L), in one aspect. Stereoscopicendoscope 302A is an example of stereoscopic endoscopic 202.

As illustrated in FIG. 3A, each stereoscopic channel in a distal end ofstereoscopic endoscope 302A, sometimes referred to as endoscope 302A,has the same component configuration. In this FIG. 3A aspect, imagecapture unit 325L (for the left stereoscopic channel) and image captureunit 325R (for the right stereoscopic channel) are symmetric withreference to a plane that intersects centerline longitudinal axis 390 ofendoscope 302A (i.e., they are positioned as mirror images of eachother). As indicated by arrow 335, the distal direction is towardstissue 303 and the proximal direction is away from tissue 303.

Light from one or more illumination channels 305 in endoscope 302Ailluminates tissue 303 in this example. While it not shown in FIG. 3A,one or more surgical instruments within the field of view of endoscope302A may also be illuminated via light from illumination channel 305.The use of an illumination channel in an endoscope is illustrative onlyand is not intended to be limiting in the various examples presented inthis description. The illumination may be provided by an illuminationsource in the endoscope or by some other apparatus that is internal orexternal to the endoscope.

Light reflected from tissue 303 and any fluorescence are received bylens assembly 301L and 301R. Lenses 304L and 304R in lens assembly 301Land 301R may include one of more optical components that direct thereceived light to sensor assembly 320L and sensor assembly 320R,respectively. In other aspects, lens assembly 301L and 301R are foldedto reduce the longitudinal length of image capture units 325L and 325R.

Light from lenses 304L and 304R passes to sensor assemblies 320L, 320R,respectively. Within sensor assemblies 320L, 320R, the light is receivedby beam splitters 331L and 331R, respectively in prism assemblies 330L,330R. In one aspect, each of beam splitters 331L and 331R is implementedas a buried coated surface 331L, 331R. As explained more completelybelow, the coating or coatings on each of coated surface 331L, 331R areselected to provide a particular functionality. Coated surface 331L,331R reflects a first portion of the light received from lens assembly301L, 301R and transmits a second portion of the received light. Fordifferentiation of features of tissue 303, the coated surfacedistinguishes between differences in polarization of the light receivedfrom lens assembly 301L, 301R. In still other aspects, the coatedsurface includes notch filters that also reflect portions of thereceived light and transmit other portions of the received light.

Irrespective of the implementation of coated surfaces 331L, 331R, beamsplitter 331L directs a first portion of the received light onto a firstimage capture sensor 310L, e.g., onto a surface 311L of image capturesensor 310L, in image capture unit 325L and transmits a second portionof the received light through beam splitter 331L. Similarly, beamsplitter 331R directs a first portion of the received light onto a firstimage capture sensor 310R, e.g., onto a surface 311R of image capturesensor 310R, in image capture unit 325R and transmits a second portionof the received light through beam splitter 331R.

In the example of FIG. 3A, light passed through beam splitters 331L and331R is received by an optional lens 350L and 350R, respectively. Lenses350L and 350R focus the received light to account for the optical pathlength to image capture sensors 315L and 315R. Lenses 350L and 350R areoptional.

The light from lenses 350L and 350R is received by reflective assemblies340L and 340R, respectively. Reflective unit 340L directs the receivedlight onto a second image capture sensor 315L, e.g., directs thereceived light onto a surface 316L of image capture sensor 315L, inimage capture unit 325L. Similarly, reflective unit 340R directs thereceived light onto a second image capture sensor 315R, e.g., directsthe received light onto a surface 316R of image capture sensor 315R, inimage capture unit 325R. In each of the aspects described herein, thelight is directed onto a surface of an image capture sensor and so forbrevity it is said that the light is directed onto the image capturesensor.

Each of reflective assemblies 340L and 340R includes a reflectivesurface 341L, 341R, e.g., a mirror surface, which reflects the receivedlight. In the example of FIG. 3A, apparatuses 340L and 340R are eachimplemented as a prism with one face having a reflective coating, or areeach implemented using total internal reflection on the hypotenuse ofthe prism. In one aspect, an angle θ formed by the intersection of aplane including reflective surface 341R and a plane including surface311R of image capture sensor 310R. and surface 316R of image capturesensor 315R is a forty-five degree angle and so the prism is referred toas a forty-five degree prism. Surface 341R of a forty-five degree prismexhibits total internal reflection when the medium proximal to surface314R is air and so surface 341R is a reflective surface.

Image capture sensors 310L and 315L are coplanar, i.e., top sensorsurfaces 311L and 316L are effectively in the same plane. Bottomsurfaces of sensors 310L and 315L are on a plane defined by a firstsurface of platform 312. Similarly, image capture sensors 310R and 315Rare coplanar, e.g., top surfaces 311R and 316R are effectively in thesame plane. Bottom surfaces of sensors 310R and 315R are on a planedefined by a second surface of platform 312. Platform 312 may becomposed of two planar parts, e.g., two ceramic parts bonded along axis390. The first surface of platform 312 is opposite and removed from thesecond surface of platform 312.

In one aspect, a first semiconductor die 317R including two imagecapture sensors 310R, 315R is mounted on a first ceramic platform. Asecond semiconductor die 317L including two image capture sensors 310L,315L is mounted on a second ceramic platform. The two ceramic platformsare then bonded together to form platform 312. Wires to the two dies317R, 317L are passed through a channel or channels in platform 312. Theuse of two image capture sensors in a die is illustrative only and isnot intended to be limiting. In some aspects, the two image sensors arein separate dies. (See FIG. 9.) The two image capture sensors may bepart of one large image capture sensor in the die, and pixels of theimage capture sensor between image capture sensor 310R and image capturesensor 315R, for example, are ignored.

In some aspects, platform 312 may not be used and the two sets of imagecapture sensors are included in a single structure configured to providethe necessary connections to power, control, and video cables. Also, theuse of the two coplanar image capture sensors in an image capture unitas shown in FIG. 3A is only illustrative and is not intended to belimiting. For example, in some aspects, more than two coplanar imagecapture sensors could be used, e.g., multiple beam splitters could beused in a line and the reflective unit placed at the proximal end of theline.

The coplanar configuration of the image capture sensors eliminates theneed for calibration to compensate for lens artifacts andre-registration of different images captured by image captures sensors310R/315R (first pair) and 310L/315L (second pair). As described above,the spatial relation between the two image capture sensors within agiven pair is constant, and since the image captures sensors within agiven pair share a common lens assembly, i.e., a common front endoptical structure, spatial registration of a pair of images captured bythe two image capture sensors remains constant over time and duringchanging optical conditions, such as changing focus.

FIG. 3B is similar to FIG. 3A, except endoscope 302B is a monoscopicendoscope, and the illumination is provided by an illuminator externalto and not connected to endoscope 302B, However, in some aspects theilluminator is mounted on endoscope 302B, in endoscope 302B, elements301, 304, 320, 330, 331, 340, 341, 350 310, 311, 315, 316, and 312B ofimage capture unit 325 are the same as elements301L, 304L, 320L, 330L,331L, 340L, 341L, 350L, 310L, 311L, 315L, 316L, and 312, respectively,and so the description of these elements is not repeated, Anillumination source external to the endoscope may be used for thevarious described endoscope embodiments and aspects rather than, or inaddition to, an illumination channel inside or mounted on the endoscope.

FIG. 4A is an illustration of a portion of an image capture unit 425Athat includes a lens assembly 401A and a sensor assembly 420A. Sensorassembly 420A includes a prism assembly 430A, a reflective unit 440A,and coplanar image capture sensors 410A, 415A, in one aspect. Lensassembly 401A includes a plurality of optical elements including anoptical element that defines an optical stop 470A, sometimes referred toas stop 470A. As used herein, a stop is a plane in space having anaperture or a place in an optical path where the chief ray has a heightof zero. Light passing through stop 470A is received by a beam splitter431A in prism assembly 430A of image capture unit 425A.

A stereoscopic apparatus would include two image capture units asillustrated in FIG. 4A. However, as demonstrated above with respect toFIG. 3A, the left and right stereoscopic channels with image captureunits are symmetric, and so only a single channel and image capture unitis described to avoid duplicative description. The other channelincluding the image capture unit is symmetric across a planeintersecting the longitudinal axis of the endoscope with the channelillustrated in FIG. 4A.

Beam splitter 431A is positioned to receive light that passes throughstop 470A. Beam splitter 431A is configured to direct a first portion ofthe received light to a first image capture sensor 410A, and to pass asecond portion of the received light through the beam splitter toreflective unit 440A. In this example, beam splitter 431A in prismassembly 430A is implemented as a coated first surface. Thus, beamsplitter 431A is sometimes referred to as coated first surface 431A.Coated first surface 431A separates the received light into the twoportions.

Coated first surface 431A reflects the first portion of the receivedlight to a second surface 432A that in turn directs, e.g., reflects, thelight onto first image capture sensor 410A. Coated first surface 431Atransmits the second portion of the received light through coated firstsurface 431A to reflective unit 440A, In one aspect, the angle ofincidence of the light to coated surface 431A is less than forty-fivedegrees.

Second surface 432A is positioned so that no light other than the lightreflected by coated first surface 431A hits second surface 432A. In oneaspect, second surface 432A is a reflective surface that is implementedas one of a coated surface and a total internal reflection surface.

Reflective unit 440A includes a third surface 441A that reflects thelight received from beam splitter 431A onto a second image capturesensor 415A. In one aspect, third surface 441A is a reflective surfaceimplemented, for example, as one of a coated surface and a totalinternal reflection surface.

image capture sensors 410A and 415A are coplanar. In one aspect, a firstoptical path length from stop 470A to coated first surface 431A tosecond surface 432A to image capture sensor 410A is about equal to asecond optical path length from stop 470A through coated first surface431A to coated surface 441A to image capture sensor 415A. In anotheraspect, the first and second optical path lengths are not equal. Theunequal optical path lengths can be implemented by adjusting the spatialrelationship between a surface in reflective unit and the beam splitterin the prism assembly, as described more completely below.

Alternatively, image capture sensor 410A has a top sensor surface in afirst plane and image capture sensor 415A has a top sensor surface in asecond plane, where the first and second planes are parallel and areseparated by a known distance (See FIG. 9). In this alternativearrangement of the image capture sensors, the optical path lengths tothe two image capture sensors can be either about equal or unequal. Thetop sensor surface of an image capture sensor is the surface of theimage capture sensor that receives light from at least one opticalcomponent in the sensor assembly. Optional optics, as illustrated bylenses 350 (FIG. 3) may be used in the second optical path to sensor415A, for example.

Herein, about equal or substantially equal means that the two opticalpath lengths are equal within the tolerances associated withmanufacturing and mounting the various optical path elements. Also, thedefinition of the optical path lengths as starting at stop 470A isillustrative and is not intended to be limiting. The optical pathlengths could also be variously defined, such as with respect to adistal face of prism assembly 430A through which the received lightenters prism assembly 430A, with respect to a first element in the lensassembly, or with respect to coated first surface 431A.

In FIG. 4A, the separation of prism assembly 430A, reflective unit 440A,and image capture sensors 410A, 415A is not to scale. In actual use,prism assembly 430A and reflective unit 440A are mostly solid glass andthere is a small gap between the glass structures and image capturesensors 410A, 415A. FIG. 4B is an example of an implementation of thestructure in FIG. 4A using prisms.

FIG. 4B is an illustration of an image capture unit 425B in distal endof a stereoscopic endoscope. Image capture unit 425B includes a sharedlens assembly 401B and a sensor assembly 420B. Sensor assembly 420Bincludes a prism assembly 430B, a reflective unit 440B, and coplanarimage capture sensors 41.013, 415B, in one aspect. Sensor assembly 420Bis positioned to receive light that passes through lens assembly 401B.The other image capture unit (not shown) includes components equivalentto those shown in FIG. 4B. The two image capture units are symmetricacross a plane that intersects the longitudinal axis of the stereoscopicendoscope in a way equivalent to that shown in FIG. 3A.

Light from an illuminator is reflected by tissue, or fluorescenceoriginating from tissue is represented by plane 403B. The light enterslens assembly 401B through a window 481 and then passes through a firstset of lenses 482, a prism 483, a second set of lenses 484, and anaperture element 470B that implements the stop.

Element 485 is optional and is illustrative of various features, eithersingly or in combination, that may be incorporated into the various lensassembly embodiments at various positions. In one aspect, element 485represents a window for notch filters to block fluorescence excitationwavelengths. In another aspect, element 485 represents a focusingelement.

In another aspect, element 485 represents a plano-plano element. In yetanother aspect, element 485 represents a structure that includes liquidcrystal cells that function as a variable focus lens when a controlvoltage is applied to the liquid crystal cells. The control voltagedynamically changes the refractive index profile of the material thelight passes through. The liquid crystal cells can adjust the focus to adesired focal distance from infinity to ten centimeters with goodoptical performance. A liquid crystal cell having these performancecharacteristics is available as the LensVector™ AutoFocus element fromLensVector, Inc. of Mountain View, Calif., US. (LENSVECTOR is atrademark of LensVector, Inc. in the United States.)

In some aspects, element 485 represents one or more fillers topreprocess the light that is received by the other elements in the lensassembly. Alternatively, the filter or filters can be attached to thedistal end surface of the endoscope, or the filter or filters may beinserted between aperture element 470B and prismatic structure 460.

In this aspect, prismatic structure 460 is an integral prismaticstructure (two prisms or more are joined without any gaps between theprisms) that includes a prism assembly 430B with beam splitter 431B anda reflective unit 440B with a reflective surface 441B. In one aspect,prismatic structure 460 is designed with a material with a particularrefractive index and prism geometry to achieve the desired imageseparation onto the two image capture sensors. The design of prismaticstructure 460 starts out as a multi-face prism and is then optimized toaccommodate the sensor spacing, and the desired maximal outer tubediameter of the stereoscopic endoscope, which restricts the prismgeometry.

In the aspect illustrated in FIG. 4B, prism assembly 430B includes apentaprism with at least two coated surfaces 431B and 432B.Alternatively, surface 432B may not be coated and may use total internalreflection. A beam splitter, i.e., coated first surface 431B, separatesthe received light into the two portions. Coated first surface 431Breflects the first portion of the received light to second surface 432Bthat in turn directs, i.e., reflects, the light onto first image capturesensor 410A. Coated first surface 431B transmits the second portion ofthe received light through surface 431A to reflective unit 440B. Prism450 is optional and is provided to facilitate handling and mounting ofprismatic structure 460.

The pentaprism is a reflecting prism used to deviate a beans of light byninety degrees. The light beam reflects inside the pentaprism twice,which allows the transmission of the light beam through a right anglewithout producing an inverted or reversed image. Typically, a pentaprismhas a periphery made up of five sides. A pentaprism is a constantdeviation prism in that the pentaprism deviates the optical path throughthe same angle irrespective of the orientation of the pentaprism to theoptical path. Herein, a pentaprism may be bonded or otherwise jointed toor positioned with respect to other prisms or optics.

In one aspect, coated first surface 431B is a multi-layer buried coatedsurface. The surface is coated with a multilayer coating, e.g., amulti-layer dichroic coating, metallic coating, or a rugate coating,which has the desired reflective and transmission properties, asdiscussed more completely below. The angle of incidence of the lightwith surface 431B is less than forty-five degrees, in one aspect,although this is not a requirement. Second surface 432B is positioned sothat no light other than the light reflected by coated first surface431B hits second surface 432B.

Distal end surface 433, sometimes referred to as a distal face, is theend of prismatic structure 460 and of prism assembly 43013 through whichthe light received from lens assembly 401B enters. In one aspect, first,second, and third surfaces 431B, 432B and 441B are positioned so that afirst optical path length from distal end surface 433 to image capturesensor 410B is about, equal to a second optical path length from distalend surface 433 to image capture sensor 415B. In another aspect, surface441B is positioned so that the two optical path lengths are not equal bya small amount.

Reflective unit 440B, in one aspect, is a prism with a surface 441Bhaving a reflective coating or surface 441B is used at such an angle toafford reflection via total internal reflection. In one aspect, an angleformed by the intersection of a plane including reflective surface 441Band a plane including a top sensor surface of image capture sensor 415Bis a forty-five degree angle and so the prism is referred to as aforty-five degree prism. In this example, prismatic structure 460 is anintegral structure formed by gluing three parts together using knowntechniques. As noted previously, the three parts are designed withmaterials, refractive index, and prism geometry to achieve the desiredimage separation on image capture sensors 410B and 415B. The use ofthree parts is illustrative only and is not intended to be limiting. Forexample, prismatic structure 460 could be an integral structure formedusing two parts.

Image capture sensors 410B and 415B are coplanar and in one aspect aremounted on a platform on which the coplanar image sensors for the otherlens assembly are also mounted (See FIG. 3A). In some aspects, the twoimage capture sensors are included in a single structure.

Stop 470B is placed or formed near or inside prismatic structure 460 toreduce the size of prismatic structure 460 and to reduce the angle ofincidence of corner rays onto image capture sensors 410B, 415B. Stop470B may be positioned either distally (as shown) or proximally ofelement 485

For the example aspects shown in FIG. 4B, middle element 483 in lensassembly 401B may be a prism. In some aspects this prism is positionedto create a folded optical path to create an endoscope with, forexample, a field of view oriented at 30 degrees to the endoscope'slongitudinal axis. This fold happens in a plane perpendicular to theplane of image capture sensors 410B, 415B.

Referring again to FIGS. 2, 3A, 3B, 4A, and 4B, various aspects of animage capture unit are shown and described. The image capture unitincludes a lens assembly that receives light from an object to beimaged. The front end optical path through the lens assembly may bestraight or may be folded to provide various angled fields of view forthe endoscope. Light from the lens assembly travels to a sensorassembly. Various optional optical components (apertures, filters,focusing elements, and the like) may be placed in the lens assembly, oras design limits allow, such optical components may be inserted in anoptical path or paths of the sensor assembly.

In one aspect, the sensor assembly includes two or more coplanar imagecapture sensors. These coplanar image capture sensors are arrangedlengthwise along the endoscope. For example, with two coplanar imagecapture sensors, one of the image capture sensors is positionedrelatively closer to the lens assembly (towards the endoscope's distalend), and one of the image capture sensors is positioned relativelyfarther away from the lens assembly (away from the endoscope's distalend). Thus, the image capture sensors are in a plane generally parallelto the endoscope's centerline longitudinal axis. In one aspect the imagecapture sensors are both formed as image capture regions (e.g., CMOSimage capture regions; alternatively, CCD regions may be used) on thesame semiconductor substrate. This single substrate therefore has alength at least about twice its width, and so it can be positionedwithin the endoscope for space use efficiency, since the sensor regionsare arranged lengthwise in the endoscope. Alternatively, the imagecapture sensors may be individually formed and positioned on a supportplatform or substrate that is arranged in the endoscope in a way similarto the single sensor substrate arrangement.

Inside the sensor assembly, light received from the lens assembly issplit into two or more beams at a beam splitter. In a two sensorimplementation, for example, one of the beams travels from the beamsplitter along one optical path to be incident on the first imagecapture sensor, and the other beam travels along another optical path tobe incident on the second image capture sensor. The optical paths in thesensor assembly are folded so that light that is incident on the imagecapture sensors is generally perpendicular to light received from thelens assembly. In some aspects, the optical paths in the sensor assemblyare arranged to be substantially equal length so that the lens assemblyaffects the images captured on the optical sensors in substantially thesame way. Various combinations of reflective surfaces, either coatedsurfaces or total internal reflection surfaces may be used to define theimage capture assembly optical path geometries. In some aspects, prismsare used to simplify reflective surface alignment requirements for theimage capture sensors.

In some aspects a single image capture unit is used in an endoscope, andso the endoscope has a monoscopic capability. In other aspects, however,two image capture units are used in an endoscope to provide stereoscopiccapability. one image capture unit provides left stereoscopic imagecapability and the other image capture unit provides right stereoscopicimage capability. In some aspects, the image capture sensors in the twoimage capture units are oriented back-to-back so that the sensors aregenerally towards the center of the endoscope. This arrangement allowsan efficient use of lateral space in an endoscope, since the front endobjective paths for the two image capture units are positioned toprovide good stereoscopic separation and the image capture sensorcircuitry can be consolidated. In some aspects the back-to-back imagecapture sensors are supported by a single platform or substrate toprovide further space savings and better optical alignment (e.g., thetwo image capture units can be aligned with one another prior toincorporation into the endoscope). The back-to-back image capturesensors may be positioned generally along the endoscope's centerlinelongitudinal axis, although in some aspects they may be offset from theendoscope's centerline longitudinal axis due to, for example, otherdistal end endoscope features that may be used. Alternatively, two imagecapture units may be positioned so that the coplanar image capturesensors are facing one another, or so that the coplanar image capturesensors for one image capture unit are coplanar with the coplanar imagecapture sensors of the other image capture sensor (i.e., all imagecapture sensors are coplanar, in some instances on the same substrate).Various other orientations of the image capture sensor planes betweenthe two image capture units may be used. It should also be understoodthat although the one or more image capture units are generallydescribed as being at the endoscope's distal end, in some aspects theone or more image capture units may be at the endoscope's proximal end.

As described in more detail below, the compact dual imaging imagecapture unit optical geometry allows two relatively large image sensorsto be placed in an endoscope for each desired optical channel, and theimage capture unit feature that splits the incoming light into two ormore beams allows many different visual display features to be presentedto a person using an imaging system. For example, if two image captureunits are used, two sets of precisely aligned stereoscopic images may becaptured, with one stereoscopic image set having the characteristicscaptured from the first light beam and the other stereoscopic image sethaving the characteristics captured from the second light beam.

Enhanced Feature Differentiation

In images of a scene from a surgical site, one problem sometimesencountered is saturation when trying to enhance the images presented tothe surgeon on surgeons control console 250. One reason is that surgicalinstruments in the field of view typically reflect more light thantissue. Another issue is identifying features of interest, e.g., nerves,diseased tissue, etc., in the scene that may not be directly on thesurface of the tissue.

In one aspect, the imaging of the scene is enhanced by using prismaticstructure 460 and coplanar image capture sensors 410B, 415B to gatheradditional information about the way the illumination interacts withtissue and other objects in the field of view. in particular, thepolarization state of the illumination light is perturbed differently bythe tissue surface, subsurface structures in the tissue, and surgicalinstruments.

There are at least two ways to implement the enhanced featuredifferentiation capability on the illumination side. Either theillumination can be predominantly polarized or predominantlyunpolarized. As is known to those knowledgeable in the field, light isnever fully polarized, and so predominantly polarized means polarized tothe extent that is required for differentiation, e.g., ideally betterthan 1,000:1 although lower contrast ratios may suffice. Similarly,predominantly unpolarized means unpolarized to the extent that isrequired.

Enhanced FeatureDifferentiation—Unpolarized Illumination

FIG. 5A is a schematic illustration of a distal end of a stereoscopicendoscope 502A with an image capture unit 525L, 525R and an illuminationchannel 505, which provides unpolarized light from an illuminator. Asindicated by arrow 535, the distal direction is towards tissue 503 andthe proximal direction is away from tissue 503.

Each image capture unit 525R, 525L includes a lens assembly 501R, 501Land a sensor assembly 520R, 520L. Sensor assembly 520R, 520L ispositioned to receive light passed through lens assembly 501R, 501L.Each sensor assembly 520R, 520L includes a prism assembly 530R, 530L, areflective unit 540R, 540L, and coplanar image capture sensors (510R,515R), (510L, 515L), in one aspect. Stereoscopic endoscope 502A issometimes referred to as endoscope 502A.

Some of the illumination from illumination channel 505 is reflected atthe surface of tissue 503. While it is not shown in FIG. 5A, some of theillumination from illumination channel 505 also may be reflected bysurgical instruments within the field of view of endoscope 502A. Sometissues impart a degree of polarization in the reflected light fromthose tissues.

In the following description, the optical paths in the right channel ofstereoscopic endoscope 502A are described. The optical paths through theleft channel of stereoscopic endoscope 502A are equivalent to those inthe right channel due to the symmetry of endoscope 502A, and so in thefollowing description, the left channel reference numeral is includedwithin parentheses following the description of an element in the rightchannel. his indicates that the description is also applicable to thecorresponding element in the left channel.

The reflected light from tissue 503, polarized and unpolarized, passesthrough lens element 504R (504L) in lens assembly 501R (501R) to stop570R (570L). The elements in lens assembly 401B (FIG. 4B) are an exampleof lens element 504R (504L). In this example, an optional quarter waveplate 580R (580L) is inserted in the optical path between stop 570R(570L) and the distal end surface of image capture unit 525R (525L). Inanother aspect (not shown), quarter wave plate 580R (580L) is riotincluded in the optical path.

The light that passes through stop stop 570R (570L) is received bysensor assembly 520L (520R) and enters a prism assembly 530R (530L) thatincludes beam splitter 531R (531L). Beam splitter 531R (531L) is apolarization based beam splitter, in this aspect. Beam splitter 531R(531L) is configured to reflect a first portion of the received light ona basis of a polarization state of the received light and to transmit asecond portion of the received light on the basis of a polarizationstate of the received light. In one aspect, beam splitter 531R (531L) isa coated surface.

For example, unpolarized light entering prism assembly 530R (530L) isevenly divided between first image capture sensor 510R (510L) and secondimage capture sensor 515R (515L) by beam splitter 531R (531L). Lightlinearly polarized entering prism assembly 530R (530L) is reflected ortransmitted by beam splitter 531R (531L) according to the relativeorientation of the polarization of the light and the coating orientationof surface 531R (531L). In the orthogonal cases, all the polarized lightis directed to either first image capture sensor 510R (510L) or secondimage capture sensor 515R (515L). In one aspect, the angle of incidenceof the light to coated surface 531R (531L) is less than forty-fivedegrees.

Coated first surface 531R (531L) reflects the first portion of thereceived light to a second surface 532R (532L) that in turn directs,e.g., reflects, the light onto first image capture sensor 510R (510L).Second surface 532R (532L) can be either a coated surface or a totalinternal reflection surface. Second surface 532R (532L) is positioned sothat no light other than the light reflected by coated first surface531R (531L) hits second surface 532R (532L).

Coated first surface 531R (531L) transmits the second portion of thereceived light through surface 531R (531L) to reflective unit 540R(540L). Specifically, the light transmitted through coated first surface531R (531L) is received by a third surface 541R (541L) of reflectiveunit 540R (540L) that in turn directs, e.g., reflects, the light ontosecond image capture sensor 515R (515L). Surface 541R (541L) can beeither a coated surface or a total internal reflection surface.

In one aspect, prism assembly 530R (530L) and reflective unit 540R(540L) are included in a prismatic structure. The prismatic structure,in this aspect, is equivalent to prismatic structure 460 (FIG. 4B) withburied coated surface 431B having a buried multi-layer polarizationselective layer, as described more completely below. Thus, thedescription of prismatic structure 460 is applicable to the prismaticstructure used in the aspect of FIG. 5A.

Coated first surface 531R (531L) is, for example, a buried multi-layerpolarization beam splitter. Such layers are known to those knowledgeablein the field, and are commonly used in polarizing beam splitter cubes.These dielectric film based coatings are usually buried and suchcoatings are used in one aspect. Alternatively, a polarization basedbeam splitter may be constructed with materials from Moxtek®Incorporated of Orem Utah, a Polatechno Co. Ltd of Japan company, as aPBF02 polarizing beam splitter. (MOXTEK is a registered U.S. trademarkof Moxtek® Incorporated of Orem Utah.)

Image capture sensors 510R (510L) and 515R (515L) are coplanar. In oneaspect, a first optical path length from stop 570R (570L) to coatedfirst surface 531R (531L) to second surface 532R (532L) to image capturesensor 510R (510L) is about equal to a second optical path length fromstop 570R (570L) through coated first surface 531R (531L) to thirdsurface 541R (541L) to image capture sensor 515R (515L). Also, thedefinition of the optical path lengths as starting at stop 570R (570L)is illustrative and is not intended to be limiting. The optical pathlengths could also be variously defined, such as with respect to adistal face of prism assembly 530R (530L) through which the receivedlight enters prism assembly 530R (530L), with respect to a first elementin lens assembly 501R (501L), or with respect to coated first surface531R (531L).

Coplanar image capture sensors 510R (510L) and 515R (515L) have a commonoptical path length through the front end optical structure in lensassembly 501R (501L) and about the same optical path length to eachimage capture sensor in sensor assembly 520R (520L). One image capturesensor 510R (510L) captures an image comprised of light reflected bypolarization beam splitter 531R (531L). Other image capture sensor 515R(515L) captures an image comprised of light transmitted by polarizationbeam splitter 531R (531L)

image capture unit 525R (525L) images two polarization states (eitherorthogonal linear states, or with quarter wave plate 580R (580L), leftand right circularly polarized states). In this case, the imagesacquired by image capture unit 525R (525L) provide information based onthe preferential polarization imparted by the structures in the tissueitself. Image capture unit 525R (525L) can only capture the relativestrength of two orthogonal components of the polarization state of thereceived light (not the entire polarization nature of the light.)However, this is sufficient to provide useful information.

For example, light is preferentially polarized when the light isspecularly reflected from a surface. The degree of the polarization (andthe state of the polarization) depends on the illumination angle and thesurface reflectance properties of the surface.

In a clinical setting, this enables reduction in some specularreflections. The reflected light that enters image capture unit 525R(525L) is separated on the basis of the linear polarization and twoimages are captured. The reflections off shiny tools in the capturedimages can be identified and reduced by executing software, because suchreflections appear in one captured image and not the other capturedimage. This process works because the light reflected by the shiny toolsis partially polarized. This process results in the reduction ofspecular reflections in the image presented to the surgeon onstereoscopic display 251.

In one aspect, dual image enhancement module 240R (240L) (FIG. 2)processes the images captured by image capture sensors 510R (510L) and515R (515L) (FIG. 5A.) to reduce the reflection from surgicalinstruments. Information from pixels in the second image captured byimage capture sensor 515R (515L) is used to modify pixels in the firstimage captured by image capture sensor 510R (510L), e.g., a percentageof the pixel value in the second image is subtracted from thecorresponding pixel value in the first image, to further reduce thebrightness of pixels representing the surgical instrument. In oneaspect, the percentage is empirically determined. This is possiblebecause the two images have the same front end optical structure and areregistered spatially relative to each other and are registeredtemporally relative to each other.

Additionally, some tissues impart a degree of polarization in the lightreflected by the tissue, for example, long stringy tissues may impartpolarization to the reflected light. In this aspect, dual imageenhancement module 240R (240L) (FIG. 2) processes the pixels in thesecond image captured by image capture sensor 515R (515L) from thereceived polarized light, and may, for example, false color the pixels.The false colored image is then combined with the first image from imagecapture sensor 510R (515L) and the combined image is sent tostereoscopic display 251 for viewing by the surgeon. In this context,“false color” is a good thing; it enables the particular pixels ofinterest to be visually salient relative to the background.

In addition, or alternatively, dual image enhancement module 240R (240L)(FIG. 2) processes the pixels in the second image captured by imagecapture sensor 515R (515L) from the received polarized light, and thendetermines corresponding pixels in the first image captured by imagecapture sensor 5 l0R (510L). Dual image enhancement module 240R (240L)changes the properties of the corresponding pixels in the first image,e.g., makes them more transparent and then combines the first and secondimages. The combined image is sent to stereoscopic display 251 forviewing by the surgeon. Since tissue that did not polarize the incidentlight is now more transparent, this allows the surgeon to more clearlyidentify, the tissue that polarized the incident light and so providesadditional cues for tissue differentiation. Thus, the combined imagegenerated by dual image enhancement module 240R (240L) increases thesaliency of a feature in the image based on polarization differences inthe received light, e.g., makes the feature more visible by makingoverlying tissue more transparent and by reducing other specularreflections.

Since the image capture sensors in an image capture unit are coplanar,the sets of image capture sensors in the two image capture units are ina fixed constant relationship to each other. Also, lens assemblies foreach of the image capture units are equivalent. Thus, there is no needfor active registration when processing the various images to assurethat the images viewed by the surgeon are aligned and form properstereoscopic images. This reduces the processing required of the imagesrelative to images that are captured without these characteristics.

Enhanced Feature Differentiation—Polarized Illumination

FIG. 5B is a schematic illustration of a distal end of a stereoscopicendoscope 502B with an image capture unit 525R, 525L and an illuminationchannel 505 that provides light from an illuminator, As indicated byarrow 535, the distal direction is towards tissue 503 and the proximaldirection is away from tissue 503.

Each image capture unit 525R, 525L includes a lens assembly 501R, 501Land a sensor assembly 520R and 520L. Sensor assembly 520R and 520L ispositioned to receive light passed through lens assembly 501R, 501L.Each sensor assembly 520R, 520L includes a prism assembly 530R, 530L, areflective unit 540R, 540L, and coplanar image capture sensors (510R,515R), (510L, 515L), in one aspect. Also, stereoscopic endoscope 502A issometimes referred to as endoscope 502A.

A polarizer 581 that polarizes the illumination is provided in theillumination optical path length. In FIG. 5B, polarizer 581 is shown atthe distal end of stereoscopic endoscope 502A, but this illustrativeonly and is not intended to be limiting to this location. Polarizer 581represents a polarizer in the path of the illumination and can be placedat an appropriate place in that path. Alternatively, illuminationchannel 505 can deliver light from a polarized source, or polarizedlight source can be located at location 581.

In the following description, the optical paths in the right channel ofstereoscopic endoscope 502B are described. The optical paths through theleft channel of stereoscopic endoscope 502B are equivalent to those inthe right channel, and so in the following description, the left channelreference numeral is included within parentheses following thedescription of an element in the right channel. This indicates that thedescription is also applicable to the corresponding element in the leftchannel.

The light from tissue 503, polarized and unpolarized, passes throughlens element 504R (504L) and stop 570R (570L). Again, the elements inlens assembly 401B (FIG. 4B) are an example of lens element 504R (504L).In this example, optional quarter wave plate 580R (580L) (FIG. 5A)between stop 570R (570L) and the distal end surface of sensor assembly520R (520L) has been removed. However, in other aspects described below,the quarter wave plate is included in the lens assembly.

The light that passes through stop 570R (570L) is received by sensorassembly 520L (520R) and enters prism assembly 530R (530L) through adistal face. As described above, polarization beam splitter 531R (531L)in prism assembly 530R (530L) is configured to reflect a first portionof the received light on a basis of a polarization state of the receivedlight and to transmit a second portion of the received light on thebasis of a polarization state of the received light.

In this example, polarization beam splitter 531R (531L) is configured todirect one polarization state onto first image capture sensor 5l0R(510L) and to transmit, e.g., pass, the remaining light to reflectiveunit 540R (540L) that in turn directs the transmitted light onto imagecapture sensor 515R (515L). Thus, the description of prism assembly 530R(530L), polarization beam splitter 531R (531L), reflective unit 540R(540L) and image capture sensors 510R (510L) and 515R (515L) is notrepeated and the above description with respect to FIG. 5A isincorporated herein by reference.

Some of the polarized illumination from polarizer 581 reflects from thesurface of tissue 503. The light reflected on the surface hasapproximately the same polarization as the polarized illumination, e.g.,the top surface reflection retains a greater portion of the originalpolarization state of the light hitting tissue 503.

Light that is not reflected at the surface enters into tissue 503 andinteracts with features 503-1 to 503-4, which are below the surface oftissue 503 and which modify the polarization of the incident light. Thelight that enters into tissue 503 is scattered or absorbed. Some of thescattered light exits the surface of tissue 503 and appears as reflectedlight at lens element 504R (504L). Thus, some of the light that exitstissue 503 provides additional information in the captured images due tothe change in polarization. Therefore, when illuminating with polarizedlight and imaging with a polarization sensitive detector, the lightwhich has a polarization different from the polarization of theillumination, e.g., light that is de-polarized, must have interactedwith the sub-surface of tissue 503.

For example, if the illumination is linearly polarized, the light isprogressively depolarized as it enters and reflects back from subsurfacefeatures 503-1 to 503-4 of tissue 503. The reflected light from thesubsurface features 503-1 to 503-4 is essentially depolarized.

Hence, with the appropriate orientations of polarizer 581 and coatedfirst surface 531R (531L), image capture sensor 515R (515L) captures animage from light primarily reflected from the surface of tissue 503 andapproximately fifty percent of the light from subsurface features 503-1to 503-4. Image capture sensor 510R (510L) captures an image from lightthat was primarily reflected from subsurface features 503-1 to 503-4 anddoes not capture a significant portion of the light from surface 503.Sensor assemblies 520R (520L) can only capture the relative strength oftwo orthogonal components of the polarization state of the receivedlight (not the entire polarization nature of the light.) However, thisis sufficient to provide useful information.

In particular, the use of polarized illumination in conjunction withpolarization sensitive imaging enables one to selectively reduce thesurface layer from the tissue in the imaging process. In this aspect,dual image enhancement module 240R (240L) (FIG. 2) processes the pixelsin the first image captured by image capture sensor 510R (510L) from thereceived light, and then determines corresponding pixels in the secondimage captured by image capture sensor 515R (515L). Dual imageenhancement module 240R (240L) can, under the control of input from userinterface 262, adjust what is shown in stereoscopic display 251. Forexample to see just surface 503, image enhancement module 240R (240L)subtracts the image captured by image capture sensor 510R (510L) fromthe image captured by image capture sensor 515R (515L), The resultingimage is sent to stereoscopic display 251 for viewing by the surgeon.Alternatively, to show subsurface features 503-1 to 503-4 in response toinput from user interface 262, image enhancement module 240R (240L)scales the image captured by image capture sensor 510R (510L) to achievesimilar brightness to the image being displayed on stereoscopic display251. The resulting image is sent to stereoscopic display 251 for viewingby the surgeon.

Since surface tissue that did not change the polarization of theincident light is now more transparent, this allows the surgeon to moreclearly identify the sub-surface features that changed the polarizationof the incident light and so provides additional cues for tissuedifferentiation. Thus, the enhanced image generated by dual imageenhancement module 240R (240L) increases the saliency of a feature inthe image based on polarization differences in the received light.

Additionally, if the illumination is circularly polarized, reflectionsfrom the top surface of tissue 503 are reflected with the handedness ofthe polarization reversed. In sensor assembly 520R (520L), if thereflected light enters through a quarter wave plate (see quarter waveplate 580R in FIG. 5A) and is then separated on the bases of the linearpolarization, the top surface reflection is significantly reduced andreflections off shiny tools are also reduced. Sub surface tissue layersstill progressively depolarize the light enabling the surface layers tobe presented to the surgeon in such a way as to appear more translucentthan before. This enables the ability to better differentiate tissuelayers, tissue types, or disease states which manifest themselves belowthe surface of tissue 503. Thus, a surgeon has the ability topreferentially see “through” the top surface of tissue 503 as thereflection from that top layer can be suppressed, as previouslydescribed, by making the top layer more transparent and by reducingspecular reflections. Additionally, just the top surface of tissue 503may be seen as described above.

Examples of tissue features this technique may accentuate may includeendometriosis. This technique also may enhance the ability todifferentiate nerves which have polarization signatures. Thus, invarious clinical settings, the combined image generated by dual imageenhancement module 240R (240L) increases the saliency of a feature inthe image based on polarization differences in the received light.

Enhanced Resolution and Dynamic Range

While the prior art cameras at the distal end of an endoscope providedstereoscopic color images, the cameras were limited to the resolutionand dynamic range provided by the single CCD that captured an image ineach stereoscopic channel. As with conventional cameras on astereoscopic endoscope, the resolution was limited by the number ofpixels of the CCDs and the color filter array. Since it is not practicalto increase the number of pixels on the CCDs given the limited spaceavailable in the distal end of a stereoscopic endoscope, furtherincreasing the resolution was not practical.

Similarly, the center of a scene of a surgical site is typically muchbrighter than the periphery of the scene. This could result in thecaptured image being clipped (in intensity) if the range of brightnessin the captured image exceeded the dynamic range of the CCD. Imagecapture units 625R, 625L (FIG. 6A) eliminate the problems of the priorart by providing both higher apparent resolution and higher dynamicrange as described more completely below.

FIG. 6A is a schematic illustration of a distal end of a stereoscopicendoscope 602 with an image capture unit 6251, 625R and an illuminationchannel 505 that provides light from an illuminator. As indicated byarrow 635, the distal direction is towards tissue 603 and the proximaldirection is away from tissue 603.

Each image capture unit 625R, 625L includes a lens assembly 601R, 601Land a sensor assembly 620R and 620L. Sensor assembly 620R, 620L ispositioned to receive light that passes through lens assembly 601R,601L. Each sensor assembly 620R, 620L includes a prism assembly 630R,630L, a reflective unit 640R, 640L, and coplanar image capture sensors(610R, 615R), (610L, 615L), in one aspect. Stereoscopic endoscope 602 issometimes referred to as endoscope 602.

In the following description, the optical paths in the right channel ofstereoscopic endoscope 602 are described. The optical paths through theleft channel of stereoscopic endoscope 602 are equivalent those in theright channel due to the symmetry of endoscope 602, and so in thefollowing description, the left channel reference numeral is includedwithin parentheses following the description of an element in the rightchannel. This indicates that the description is also applicable to thecorresponding element in the left channel. Also in FIG. 6A, elementswith the same reference numeral as elements in FIGS. 5A and 5B are thesame or equivalent elements to those previously described with respectof FIGS. 5A and 5B. To avoid repetition, elements with the samereference numeral are not described again in detail with respect to FIG.6A.

The light from tissue 503 passes through lens element 504R (504L) andstop 570R (570L) in lens assembly 501R (501L). The elements in lensassembly 401B (FIG. 4B) are an example of lens element 504R (504L). Thelight received by sensor assembly 620L (620R) enters prism assembly630R. Prism assembly 630R (630L) includes a beam splitter 631R (631L),e.g., a coated first surface, that reflects a first percentage of thelight received by prism assembly 630R (630L) and that passes a secondpercentage of the received light through the coated first surface, e,2.,through beam splitter 631R (631L) to reflective unit 540R (540L).

The light reflected by coated first surface 631R (631L) is received by asecond surface 632R (632L) that in turn directs, e.g., reflects, thelight onto first image capture sensor 610R (610L). Second surface 632R(632L) is, for example, one of a coated surface, and a total internalreflection surface. The light transmitted through coated first surface631R (631L) is received by a third surface 541R (541L) of reflectiveunit 540R (540L) that in turn directs, e.g., reflects, the light ontosecond image capture sensor 615R (615L).

Second surface 632R (632L) is positioned so that no light other than thelight reflected by coated first surface 631R (631L) hits second surface632R (632L). In one aspect, the angle of incidence of the light tocoated first surface 631R (631L) is less than forty-five degrees.

In one aspect, prism assembly 630R (630L) and reflective unit 540R(540L) are included in a prismatic structure with a pentaprism thatincludes prism assembly 630R (630L). The prismatic structure in thisaspect is equivalent to prismatic structure 460 (FIG. 4B) with buriedcoated surface 431B configured to reflect a first percentage of thereceived light and to pass through the coated surface a secondpercentage of the received light. Thus, the description of prismaticstructure 460 is applicable to the prismatic structure used in theaspect of FIG. 6A.

In one aspect, the first and second percentages are about equal. Inanother aspect, the first and second percentages are different. Thus,prism assembly 630R (630L) includes a beam splitter implemented as acoated first surface 531R (531L) that separates the received light intothe two portions—(i) a first percentage of the received light sometimesreferred to as a first portion, and (ii) a second percentage of thereceived light, sometimes referred to as a second portion.

Image capture sensors 610R (610L) and 615R (615L) are coplanar. In oneaspect, a first optical path length from stop 570R (570L) to coatedfirst surface 531R (531L) to second surface 632R (632L) to image capturesensor 610R (610L) is about equal to a second optical path length fromstop 570R (570L) through coated first surface 631R (631L) to a thirdsurface 541R (541L) to image capture sensor 615R (615L). Again, thedefinition of the optical path lengths as starting at stop 570R (570L)is illustrative and is not intended to be limiting. The optical pathlengths could also be variously defined, such as with respect to adistal face of prism assembly 630R (630L) through which the receivedlight enters prism assembly 630R (630L), with respect to a first elementin lens assembly 601R (601L), or with respect to coated first surface631R (631L).

Thus, coplanar image capture sensors 610R (610L) and 615R (615L) have acommon optical path length through the front end optical structure inlens assembly 601R (601L) and about the same optical path length to eachimage capture sensor in sensor assembly 620R (620L). One image capturesensor 610R (610L) captures an image from the first portion of the lightreceived by sensor assembly 620R (620L). Other image capture sensor 615R(615L) captures an image from the second portion of the light receivedby sensor assembly 620R (620L). As described more completely, below, inone aspect, each of image capture sensors 610R (610L) and 615R (615L) isa color sensor with a color filter array. In another aspect, the colorfilter array is removed from one of the color sensors and the sensorfunctions as a monochrome sensor. The color filter array of the othercolor sensor is modified for the number of color components received bythat sensor.

Enhanced Resolution—Example One

In one aspect, coated first surface 631R (631L) of prism assembly 630R(630L) is configured to reflect and to transmit about equal portions ofthe light received by prism assembly 630R (630L) from lens assembly 601R(601L), i.e., the first and second percentages are about equal. Whenbeam splitter 631R (631L) reflects and transmits about equal portion oflight, the beam splitter is referred to as a balanced beam splitter.Each of image capture sensors 610R (610L) and 615R (615L) is a colorsensor with a color filter array in this aspect. The color filter arrayis a Bayer color filter array. Thus, the two Bayer pattern image capturesensors are looking through the same optics at the same scene. Here, aBayer pattern image capture sensor is a single chip sensor, or part of asingle chip, that includes a Bayer color filter array. As noted above,coplanar image capture sensors 610R (610L) and 615R (615L) have a commonfront end optical structure and about the same optical path length toeach sensor.

When prism assembly 630R (630L) and reflective unit 540R (540L) arearranged so that the color images captured by both Bayer pattern imagecapture sensors 610R (610L) and 615R (615L) are the s color imagecaptured by image capture sensor 610R (610L) is the same scene as thecolor image captured by image capture sensor 615R (615L). Thus, withsome static calibration, each point in space in the scene is representedby two pixels one pixel in the color image captured by image capturesensor 610R (610L) and one pixel in the color image captured by imagecapture. sensor 615R (615L).

Capturing two pixels for each port hr space in e scene has severaladvantages over a normal color image that has one pixel for each pointin space in the scene. For example in display 251, pixels that are eachbased on two pixels for a point in space have reduced noise level andgreater apparent resolution. Each output pixel to stereoscopic display251 from dual image enhancement, module 240R (240L) in centralcontroller 260 is based on sampling of two pixels, one from each of theimages captured by Bayer pattern image capture sensors 610R. (610L) and615R (615L).

The sampling of two input pixels by dual image enhancement module 240Rallows imaging of smaller features than is possible if only an imagefrom a single image capture sensor is processed by central controller260. Thus, the apparent resolution in the image viewed on stereoscopicdisplay 251 is greater than the resolution in an image viewed onstereoscopic display 251 based on an image captured by a single imagecapture sensor. Here, the resolution of an image sent to stereoscopicdisplay 251 can be higher than that of the image from a single imagecapture sensor and so is said to have greater apparent resolution.

Enhanced Resolution—Example Two

In another aspect, coated first surface 631R (631L) of prism assembly630R (630L) is still configured to reflect and transmit about equalportions of the light received by prism assembly 630R, i.e., the firstand second percentages are about equal. Each of image capture sensors610R (610L) and 615R (615L) is a color sensor with a Bayer color filterarray. Again, coplanar image capture sensors 610R (610L) and 615R (615L)have a common front end optical structure and about the same opticalpath length to each sensor. However, surfaces 631R. (631L) and 632R(632L) of prism assembly 630R (630L) are tilted slightly, withoutchanging the total optical path length, so that the image captured byimage capture sensor 610R (610L) is offset from the image captured byimage capture sensor 615R (615L) by one-half pixel.

In this case, two Bayer pattern image capture sensors are “looking”through the same optics at the same scene. However, the two capturedimages are offset from one another by a half pixel. FIG. 6B is aschematic illustration of the offset for a block of pixels 691 in thefirst image and a corresponding block of pixels 695 in the second image.Each pixel in block of pixels 691 is represented by a square having adashed line perimeter and the color of the pixel is given by the firstletter of the color followed by the number 2 at the center of thesquare. Each pixel in block of pixels 695 is represented by a squarehaving a solid line perimeter and the color of the pixel is given by thefirst letter of the color followed by the number 1 at the center of thesquare. In the example, the Bayer color filter array is ared-green-blue-green (RGBG) array.

In one aspect, dual image enhancement module 240R (240L) in centralcontroller 260 interpolates the pixels in the two images to create acolor image with higher spatial resolution for the image sent tostereoscopic display 251. The interpolation is equivalent to that donein a three CCD color camera, except unlike the three CCD color camera,two color images are used. The larger number of pixels available for thesampling enhances the apparent resolution and provides improvedsignal-to-noise performance.

Enhanced Resolution—Example Three

In yet another aspect, coated first surface 631R (631L) of prismassembly 630R (630L) is configured to reflect a first color component inthe light received by prism assembly 630R from lens assembly 601R (601L)and to transmit other color components in the received light. In thisaspect, image capture sensors 610R (610L) and 615R (615L) are not each acolor sensor with a Bayer color filter array.

Rather, image capture sensor 610R (610L) is a monochrome sensor, e.g., acolor sensor with the color filter array removed. For purposes ofillustration, the light received by sensor assembly 620R (620L) is takenas having a plurality of visible color components—a first visible colorcomponent, a second visible color component, and a third visiblecomponent. Image capture sensor 615R (615L) is a color sensor with acolor filter array for two of the three visible color components, e.g.,the number of visible color components in the plurality of visible colorcomponents minus one. As noted above, coplanar image capture sensors610R (610L) and 615R (615L) have a common front end optical structureand about the same optical path length to each sensor.

In this aspect, prism assembly 630R (630L) and reflective unit 540R(540L) are arranged so that the images captured by both image capturesensors 610R (610L) and 615R (615L) are the same scene, but the firstimage is monochrome and the second image is multi-color. For example,the monochrome image represents the green color component image, and themulti-color image is a red color component and a blue color componentimage. In this case, color filter array of image capture sensor 615R(615L) is a red and blue checkerboard pattern.

The ratio of green pixels to red and blue pixels is the same as in aBayer color filter array, but there are twice as many pixels of eachcolor compared to a prior art single color sensor. Thus, the resolutionof the green pixels is generated using full spatial sampling by dualimage enhancement module 240R (240L) in central controller 260, whilethe resolution of the red and blue pixels is generated using reducedspatial sampling, relative to the green pixels, by dual imageenhancement module 240R (240L). Nevertheless, the spatial sampling forthe red and blue pixels in this aspect is the same as the spatialsampling for the green pixels in the prior art single color sensor. Thisenhanced spatial sampling also provides enhanced apparent resolution andimproved signal to noise ratio.

Enhanced Dynamic Range

Typically, surgical site images are similar and have a bright centralregion and a darker peripheral region. Also, reflections from somesurgical instruments are much brighter than reflections from tissue.This results in differences in the range of brightness values that arecaptured by the image capture sensor in a prior art endoscope. If thebrightness values are beyond the dynamic range of the image capturesensor, the values for the pixels are clipped, i.e., set to the highestvalue of the image capture sensor.

Suppose a scene has a brightness range from 0 to 400. If this scene isimaged to a sensor having a dynamic range from 0 to 100, values over 100are clipped. For example if the brightness value for a pixel should be400, the brightness value captured for that pixel is 100. Brightnessinformation is lost for parts of the scene having a brightness largerthan 100, because all brightness values greater than 100 are clipped andset to 100. If the gain for the image capture sensor is adjusted to0.25, the high brightness values are not clipped, but all scenebrightness values lower than four are mapped to zero. For this case, thehigh brightness values are not clipped, but information in the darkerparts of the scene is lost. Image capture unit 625R (625L) provides asolution to this problem by preserving both the high brightnessinformation and more of the low brightness information relative to theprior art systems. A brightness range of 0 to 400 is illustrative only.In an actual surgical scene, the brightness variation may be many ordersof magnitude. Thus, the dynamic range of the surgical scene is greaterthan the dynamic range of the image capture sensor. Thus, a single imagecapture sensor cannot capture the dynamic range of the surgical scene.

Coated first surface 631R (631L), e.g., the beam splitter, of prismassembly 630R (630L) in sensor assembly 620R (620L) is configured toreflect and to transmit different portions of the light received byprism assembly 630R (630L), i.e., the first and second percentagesdefined above are different. Beam splitter 631R (631L) is a dynamicrange adjusted beam splitter in this aspect. A dynamic range adjustedbeam splitter reflects M % of the received light and passes N % of thereceived light, where M % is different from N %. Here, M and N arepositive numbers. In one aspect, M % plus N % is equal to about onehundred percent. The equality may not be exact due to light losses anddue to tolerances of the various parts of sensor assembly 620R (620L).

In this aspect, each of image capture sensors 610R (610L) and 615R(615L) is a color sensor with a color filter array. The color filterarray is a Bayer color filter array in one aspect. Thus, the two Bayerpattern image capture sensors 610R (610L) and 615R (615L) are lookingthrough the same optics at the same scene. Here, a Bayer pattern imagecapture sensor is a single chip sensor, or part of a single chip, thatincludes a Bayer color filter array. As noted above, coplanar imagecapture sensors 610R (610L) and 615R (615L) have a common front endoptical structure and about the same optical path length to each sensor.For a pair of coplanar image capture sensors 610R (610L) and 615R (615L)with known dynamic ranges, e.g., the gains of image capture sensors 610R(610L) and 615R (615L) are adjusted to correspond to the configurationof beam splitter 631R (631L).

Typically, the selection of properties of the coating for first surface631R (631L) considers the brightness of the image captured and/or thebrightness of a portion of the image captured, the possible dynamicranges of image capture sensors 610R (610L) and 615R (615L), and thecapabilities of the imaging pipeline. As used here, the imaging pipelineis the portion of central control system 260 (FIG. 2) that processes thecaptured images and generates output images for stereoscopic display251. The imaging pipeline may be part of CCU 230R (230L) or part of dualimage enhancement module 240R (240L) depending on the implementationchosen.

To assure that no information is clipped in a region of a typicalsurgical scene that is of importance, a maximum brightness Bmax of theregion is empirically determined. The region of importance can be eitherthe complete scene or a portion of the complete scene. The dynamic rangeof image capture sensor 610R (610L) is then determined, e.g., 0 toS1max. Fraction M of the received light that is reflected by the coatingon surface 631R (631L) is selected as:

M≈(S1max(Bmax)

The percentage of the received light M % that is reflected by thecoating on first surface 631R (631L) is selected so that dynamic rangeof light incident on first image capture sensor 610R (610L) is not,clipped. Thus, the brightness of pixels in high brightness regions ofthe image captured by first image capture sensor 610R (610L) are notclipped in this aspect. The percentage of the received light transmittedby the coating on surface 631R (631L) is N %, where N % is about equalto one hundred minus M %. The gain of second image capture sensor 615R(615L) is adjusted so the dynamic range of sensor 615R (615L) is fromzero to about N % times maximum brightness Bmax.

As an example, consider a scene that has a maximum brightness Bmax of400. Image capture sensor 610R (610L) has a dynamic range of 0 to 100.Thus,

M=100/400=¼

M %=25%

N =75%

Thus, the coating on first surface 631R (631L) is selected to reflectabout 25% of the received light and to transmit about 75% of thereceived light. The gain for image capture sensor 615R (615L) isadjusted to have a dynamic gain of 0 to 300.

The two images captured by sensors 610R (610L) and 615R (615L) areacquired essentially simultaneously. The light received by sensorassembly 620R (620L) is split into two images with a brightness range onsensor 610R (610L) being 0 to 100 (¼*400) and a brightness range onsensor 615R (615L) being 0 to 300 (¾*400). Pixels in the high brightnessregions of the scene are not saturated in the image captured by sensor615R (615L). At the same time, the image captured by sensor 610R (610L)preserves and accurately images the darker portions of the scene. Notethat in this example scene regions with a brightness of one may be lostif the image capture sensors do not save values less than one. However,this is only one part out of four hundred so is unlikely to be noticedby the surgeon viewing the image on stereoscopic display 251.

In some situations, better results may be obtained by intentionallyletting the sensor receiving the most light saturate at the high end.This expands the range at the low end where the other sensor is too darkto register any values.

When prism assembly 630R (630L) and reflective unit 540R (540L) arearranged so that the color images captured by both Bayer pattern imagecapture sensors 610R (610L) and 615R (615L) are the same scene withdifferent brightness ranges, each point in space in the scene isrepresented by two pixels one pixel in the color image captured by imagecapture sensor 610R (610L) and one pixel in the color image captured byimage capture sensor 615R (615L). However, the two pixels have differentbrightness values that are a linear scale of each other if thebrightness value on neither pixel has been clipped.

In this example, the maximum brightness was assumed known. However, in aclinical setting, the structure of stereoscopic endoscope is fixed andnot every scene will have the same maximum brightness. Thus, if theconfiguration just described is used in a scene with a maximumbrightness of 500, the pixels for the brightest part of the scene willbe clipped. Nevertheless, the dynamic range is still extended over theprior art solutions.

Capturing two pixels with different brightness values for each point inspace in the scene has advantages over a normal color image that has onebrightness value for each point in space. For example in display 251,pixels that are each based on two pixels for a point in space havereduced noise level for mid brightness ref-lions and the dynamic rangeis extended compared to the prior art. These advantages are in additionto the spatial resolution advantages described above.

Each output pixel to stereoscopic display 251 from dual imageenhancement module 240R (240L) in central controller 260 is based onsampling of two pixels, one from each of the images captured by Bayerpattern image capture sensors 610R (610L) and 615R (615L). As notedabove, the sampling of input two pixels by dual image enhancement module240R (240L) allows imaging of smaller features than is possible if onlyan image from a single image capture sensor is processed by centralcontroller 260. In addition, in one aspect, a tone mapping process indual image enhancement module 240R (240L) maps regions of the acquiredimages to preserve contrast and to match the dynamic output range ofstereoscopic display 251. The tone mapping maps the brightness from thecaptured images to the displayable range while preserving the imagecontrast and color appearance important to best represent the originalscene content. The tone mapping is similar to prior art techniquesexcept that it is enhanced by sampling two pixel values for the samepoint in space and time instead of one pixel value sequentially.

In the above example, brightness values from 0 to 400 are mapped to thedynamic output range (contrast ratio) of stereoscopic display 251 by thetone mapping. For pixels that are not saturated, the brightness value inthe image captured by sensor 615R (615L) is three times the brightnessvalue in the image captured by sensor 610R (610L) for the same point inspace. The tone mapping process utilizes this relationship indetermining the brightness for a pixel in an image output tostereoscopic display 251. Based on information on the surroundingpixels, tone mapping works on individual groups/regions of pixels tomaintain the contrast and thus reduces the noise level and the dynamicrange.

The prior art brightness range 0 to 100 has been extended to abrightness range of 1 to 400 for a gain in overall dynamic rangeperformance. Combining the two images and taking advantage of the doublesampling further enables reductions in noise in the middle brightnessregions along with the increased dynamic range overall.

In the aspect of FIG. 6A and each of the examples described above withrespect to FIG. 6A, the image capture sensors in a stereoscopic channelare coplanar. This is illustrative only and is not intended to belimiting. For example, a first of the image capture sensors could have atop sensor surface in a first plane, and a second of the image capturesensors could have a top sensor surface in a second plane. (See FIG. 9.)The first and second planes are parallel to each other and are separatedby a known distance. The spacing between the beam splitter and thereflective surface in the reflective unit is adjusted to compensate forthe known distance so that the first optical path length to the firstimage capture sensor is about equal to the second optical path length tothe second image capture sensor. Thus, each of the aspects of FIG. 6Adescribed above are directly applicable to this configuration as the twooptical path lengths remain about equal.

Enhanced Color Performance

Currently, most imaging system use three visible color components, red,green, and blue (RGB), which is referred to as a three color primarymodel. Most cameras use a Bayer color filter array image capture sensorwith a RGB based color filter array. The three color primary model isalso used in most liquid crystal displays (LCDs) and plasma displays.

The three color primary model actually restricts the range of hues,which can be captured by a camera and presented in a display. Usingadditional color primaries enables the capture and display of moreperceptually accurate images. Sharp Electronics Corporation (Sharp)sells light emitting diode (LED) backlit LCD televisions with quad pixeltechnology. Sharp adds yellow to the conventional red, green and bluecolor filter array, enabling richer colors to be displayed.

Capturing additional color components in an image by a camera wouldenable more faithful color reproduction. However, capturing more colorcomponents would require changing the conventional Bayer red, green,green, blue (RGGB) color filter array to include the new colors such asyellow and orange, i.e., changing the color filter array to red, orange,green, yellow, green, blue (ROGYGB). However, for an image capturesensor with a given number of pixels, this six component color filterarray would reduce the spatial resolution of the camera and wouldrequire the development of new masks and dyes suitable for printing onimage sensors. The selection of the yellow or orange colors would beoptimized in concert with the display.

The use of an appropriate illuminator in combination with image captureunits 725R (725L) provides up to a six color component vector per pixel,for example, without reducing the spatial resolution and withoutrequiring the development of new masks and dyes suitable for printing onimage sensors. This capability is implemented in a number of ways asdescribed more completely below. The capability not only provides from athree to a six color component vector per pixel after de-mosaicing, butalso provides images with an extended depth of field, and also providesenhanced fluorescence imaging capability.

FIG. 7A is a schematic illustration of a distal end of a stereoscopicendoscope 702 with image capture units 725L, 725R and an illuminationchannel 705 that provides light from, for example, one of illuminator710B (FIG. 7B), illuminator 710C (FIG. 7C), illuminator 710D (FIG. 7D)and illuminator 710E (FIG. 7E). See FIG. 2 for an illustration of howthe illuminator is coupled to the illumination channel. As indicated byarrow 735, the distal direction is towards tissue 703 and the proximaldirection is away from tissue 703.

Each image capture unit 725R, 725L includes a lens assembly 701R, 701Land a sensor assembly 720R, 720L. Sensor assembly 720R, 720L ispositioned to receive light that passes through lens assembly 701R,701L. Each sensor assembly 720R, 720L includes a prism assembly 730R,730L, a reflective unit 740R, 740L, and coplanar image capture sensors(710R, 715R), (710L, 715L), in one aspect. Stereoscopic endoscope 702 issometimes referred to as endoscope 702.

In the following description, the optical paths in the right channel ofstereoscopic endoscope 702 are described. The optical paths through theleft channel of stereoscopic endoscope 702 are equivalent those in theright channel due to the symmetry of endoscope 702, and so in thefollowing description, the left channel reference numeral is includedwithin parentheses following the description of an element in the rightchannel. This indicates that the description is also applicable to thecorresponding element in the left channel. Also in FIG. 7A, elementswith the same reference numeral as elements in FIGS. 5A and 5B are thesame or equivalent elements to those previously described with respectof FIGS. 5A and 5B. To avoid repetition, elements with the samereference numeral are not described again in detail with respect to FIG.7A.

The basic structure of image capture units 725L, 725R in each of thedifferent aspects described below is presented in FIG. 7A. The spatialrelationships and alignments between prism assembly 730R (730L),reflective unit 540R (540L), image capture sensors 710R (710L), andimage capture sensors 715R (715L) are the same for each of the differentaspects considered with respect to FIG. 7A. However the coatings oncoated first surface 731R (731L) of prism assembly 730R (730L), the typeof image capture sensors, and the fillers used on the image capturesensors may change in different as aspects, as described more completelybelow.

The reflected light from tissue 703 passes through lens element 704R(704L) and stop 570R (570L) in lens assembly 701R (701L). The elementsin lens assembly 401B (FIG. 4B) are an example of lens element 704R(704L). The light that passes through lens assembly 701R (701L) isreceived by prism assembly 730R (730L) and enter beam splitter 731R(731L). Beam splitter 731R (731L) is implemented, in this aspect, as acoated first surface that reflects a first portion of the receivedlight, and passes a second portion of the received light to reflectiveunit 540R (540L).

The light reflected by coated first surface 731R (731L) is received by asecond surface 732R (732L) in prism assembly 730R (730L) that in turndirects, e.g., reflects, the light onto first, image capture sensor 710R(710L). Surface 732R (732L) can be either a coated surface or a totalinternal reflection surface. The light transmitted through coated firstsurface 731R (731L) is received by a third surface 541R (541L) ofreflective unit 540R (540L) that in turn directs, e.g., reflects, thelight onto second image capture sensor 715R (715L).

Second surface 732R (732L) is positioned so that no light other than thelight reflected by coated first surface 731R (731L) hits second surface732R (732L). In one aspect, the angle of incidence of the light tocoated first surface 731R (731L) is less than forty-five degrees.

In one aspect, prism assembly 730R (730L) and reflective unit 740R(740L) are included in a prismatic structure with a pentaprism. thatincludes prism assembly 730R (730L). The prismatic structure in thisaspect is equivalent to prismatic structure 460 (FIG. 4B) with buriedcoated surface 4319 including a plurality of notch filters, as describedmore completely below. Thus, the description of prismatic structure 460is applicable to the prismatic structure used in this aspect.

In one aspect, the first portion reflected by coated first surface 731R(731L) includes first selected wavelengths of a plurality of colorcomponents in the received light, and the second portion transmitted bycoated first surface 731R (731L) includes second selected wavelengths ofthe plurality of color components in the received light. The firstselected wavelengths of the plurality of color components in thereceived light are different from the second selected wavelengths of theplurality of color components in the received light. In one aspect,coated first surface 731Rr (731L) comprises a plurality of notchfilters. The notch filters separate the received light into the firstportion and the second portion.

Image capture sensors 710R (710L) and 715R (715L) are coplanar. In oneaspect, a first optical path length from stop 570R (570L) to coatedfirst surface 731R (731L) to second surface 732R (732L) to image capturesensor 710R (710L) is about equal to a second optical path length fromstop 570R (570L) through coated first surface 731R (731L) to a thirdsurface 741R (741L) to image capture sensor 715R (715L). Again, thedefinition of the optical path lengths as starting at stop 570R (570L)is illustrative and is not intended to be limiting. The optical pathlengths could also be variously defined, such as with respect to adistal face of prism assembly 730R (730L) through which the receivedlight enters prism assembly 630R (630L), with respect to a first elementin lens assembly 701R (701L), or with respect to coated first surface731R (731L).

Thus, coplanar image capture sensors 710R (710L) and 715R (715L) have acommon front end optical structure and about the same optical pathlength to each sensor. A first image capture sensor 710R (710L) capturesan image from the first portion of the light received by sensor assembly720R (720L). A second image capture sensor 715R (7151.) captures animage from the second portion of the received light. As described morecompletely, below, in one aspect, each of image capture sensors 710R(710L) and 715R (715L) is a color sensor with a standard color filterarray. In another aspect, the color filter array on one of the colorsensors and the sensor function is selected based on the light that isreceived by endoscope 702.

Controller 260 processes the first and second images captured by imagecapture sensors 710R (710L) and 715R (715L), respectively, and generatesan output image for stereoscopic display 251. Controller 260 generatesan N color component vector for each pixel in the output image, where Nranges from three to six. The color component vector for a pixel in theoutput image is generated from a color component vector of acorresponding pixel (or some combination of pixels) in the first imageand a color component vector of a corresponding pixel in the secondimage. Recall, that each pixel in the output image represents a point inspace in the scene captured by the image capture sensors. The pixels inthe captured images corresponding to the pixel in the output image arethe pixels representing the same point in space.

Enhanced Color Performance—Six Color Component Illumination

In a first aspect, a six color component illuminator 710B (FIG. 7B) iscoupled to illumination channel 705 (FIG. 7A). Illuminator 710B includessix laser illumination sources 710B1 to 710B6 that generate red colorcomponent R1, red color component R2, green color component G1, greencolor component G2, blue color component B1, and blue color componentB2, respectively. With laser illumination. sources, the wavelengths ofred color components R1 and R2, green color components G1 and G2, andblue color components B1 and B2 are typically 2 to 5 nanometer (nm) wideinstead of about 60 nm per color component from a conventional lightemitting diode based illuminator. The configuration of an illuminatorwith multiple different illumination sources with a mirror for source710B1 and dichroic mirrors for sources 710B2 to 710B6 is known and so isnot considered in further detail herein. See for example, U.S. patentapplication Ser. No. 12/855,905 (filed Aug. 13, 2010; disclosingSurgical Illuminator with Dual Spectrum Fluorescence), which isincorporated herein by reference.

In this aspect, coated first surface of 731R (731L) of prism assembly730R (730L) includes a plurality of notch filters. A first notch filterreflects red color component R1 in the received light and transmits redcolor component R2 in the received light. A second notch filter reflectsgreen color component G1 in the received light and transmits green colorcomponent G2 in the received light. A third notch filter reflects bluecolor component B1 in the received light and transmits blue colorcomponent B2 in the received light. The use of three notch filters isfor ease of discussion and is not intended to be limiting to threeseparate filters. In view of this description, one knowledgeable in thefield can implement a notch filter with the reflect, transmit propertiesdescribed here. The coating design would implement the notches. See forexample, the “Stopline” and “Quad-Notch” product lines of SemrockProducts, Rochester, N.Y.

Hence, coated first surface 731R (731L) reflects red color component R1in the received light, green color component G1 in the received light,and blue color component B1 in the received light to second surface 732R(732L). Second surface of 732R (732L) reflects red color component R1,green color component G1, and blue color component B1 received fromcoated first surface 731R (731L) onto image capture sensor 710R (710L).

Coated first surface of 731R (731L) transmits red color component R2 inthe received light, green color component G2 in the received light, andblue color component B2 in the received light to third surface 541R(541L) in reflective unit 540R (540L). Third surface of 541R (541L)reflects red color component R2, green color component G2, and bluecolor component B2 received from coated first surface 731r (731L) ontoimage capture sensor 715R (715L).

Each of image capture sensors 710R (710L) and 715R (715L) is a colorsensor with a color filter array in this aspect. The color filter arrayis a Bayer red, green, green, blue (RGGB) color filter array. Thus, thetwo Bayer pattern image capture sensors are looking through the sameoptics at the same scene. Here, a Bayer pattern image capture sensor isa single chip sensor, or part of a single chip, that includes a Bayercolor filter array. As noted above, coplanar image capture sensors 710R(710L) and 715R (715L) have a common front end optical structure andsubstantially the same optical path length to each sensor.

When prism assembly 730R (730L) and reflective unit 540R (540L) arearranged so that the color images captured by both Bayer pattern imagecapture sensors 710R (710L and 715R (715L) are the same scene, the colorimage captured by image capture sensor 710R (710L) has the same fullresolution as the color image captured by image capture sensor 715R(715L). Thus, each point in space in the scene is represented by twopixels with each of the two pixels having a different three colorcomponent vector. Sensor assembly 720R (720L) and consequently imagecapture unit 725R (725L) has acquired a full resolution image with sixprimary color components with no loss of light or spatial resolution.

The color component vector for each pixel output to stereoscopic display251 from dual image enhancement module 240R (240L) in central controller260 is derived from the six primary color components for this pixel.This permits driving a stereoscopic display with for example, a fourelement vector for each pixel without any loss in resolution. Dual imageenhancement module 240R (240L) samples the two captured images andapplies a color correction matrix to generate the pixel vector requiredby stereoscopic display 251.

The color component vector sent to stereoscopic display 251 can havemore than three color components to more accurately represent thespectral content of the scene. In actual use, the number of colorcomponents used in the color component vector, sometimes referred to asa color vector, would match the number of color components used by thedisplay. Sharp has shown five color component displays.

In another aspect, it is noted, for example, that when fluorescence isexcited, the fluorescence may be in the visible spectrum. For example,Fluorescein may be excited by 490 nm blue light and fluoresces mainly inthe 520 to 530 nm range. Thus, in this aspect, a color filter thatblocks transmission of the reflected illumination from the excitationlaser module, at say 490 nm, is included in coated first surface 731R(731L). The notch filter for the blue wavelengths is configured toreflect color component B1 and to pass other blue wavelengths so that,the fluorescence from the Fluorescein is passed to image capture sensor715R (715L). The notch filters and the illumination components from theilluminator for the red and green color components are selected so thatthe reflected color components R1., B1, and. G1 are captured by firstimage capture sensor 710R (710L). The remaining colors are delivered tosecond image capture sensor 715R (715L). Note that the excitation lasercomponent B1 may saturate the blue pixels of first image capture sensor710R (710L), but this is acceptable as the fluorescence from theFluorescein is captured by second image capture sensor 715R (715L)

In another aspect, a color correction module 241R (241L) in dual imageenhancement module 240R (240L) of central controller 260 uses the threecolor pixels from the first image and the three color pixels from thesecond image to generate a five color component output vector for aSharp five color display, e.g., generated a red, red complement, yellow,green, blue (RR′YGB) color component. The color correction matrix is setto the desired mapping from the six element acquisition color space tothe display's five element color space.

More specifically, color correction module 241R (241L) in dual imageenhancement module 240R (240L) of central controller 260 is coupled tothe first and second image capture sensors 710R (710L) and 715R (715L).Camera control unit 230R (230L) demosaics the first image captured bythe first image capture sensor 710R (710L), and demosaics the secondimage captured by the second image capture sensor 715R (715L). Eachpixel in the demosaiced images has a three element color vector. The twodemosaiced imaged are received by color correction module 241R (241L).The three element color vectors are combined from the two demosaicedimages to create an N-element color component vector for each pixel,where N is at least three and in this example N is six. The colorcorrection matrix in color correction module 241R (241L) then generatesan M-element color component vector for a pixel in an output image fromthe N-element color component vector of the corresponding pixel. Notethat when M is three, the processes described above that utilize the twocaptured images to enhance resolution, etc. can also be applied.

Enhanced Color Performance—White Broadband Illuminiation

In a second aspect, a white broadband illuminator 710C (FIG. 7C) iscoupled to illumination channel 705 (FIG. 7A). In one example,illuminator 710C use a Xenon lamp 710C. with an elliptic back reflectorand a band pass filter coating to create broadband white illuminationlight with little infrared content. The use of a Xenon lamp isillustrative only and is not intended to be limiting. For example, ahigh pressure mercury arc lamp, other arc lamps, or other broadbandlight sources may be used.

In this aspect, coated first surface of 731R (731L) of prism assembly730R (730L) includes a filter 790F (FIG. 7F) that includes a pluralityof notch filters. A first notch filter 791F reflects a first part R1 ofa red color component R in the received light and transmits a complementR1′ of first part R1 of red color component R in the received light. Asecond notch filter 792F reflects a first part G1 of a green colorcomponent G in the received light and transmits a complement of firstpart G1 of green color component G in the received light. A third notchfilter 793F reflects a first part B1 of a blue color component B in thereceived light and transmits a complement B1′ of first part B1 of bluecolor component B in the received light. The use of three notch filtersis for ease of discussion and is not intended to be limiting to threeseparate filters. The three separate notch filters can be viewed as asingle multi-notch filter with the reflect, transmit propertiesdescribed here.

Hence, coated first surface of 731R (731L) reflects red color componentR1 in the received light, green color component G1 in the receivedlight, and blue color component B1 in the received light onto secondsurface 732R (732L). Coated surface of 732R (732L) reflects allwavelengths received from coated first surface 731R (731L) onto imagecapture sensor 710R (710L).

Coated first surface of 731R (731L) transmits red color component R1′ inthe received light, green color component G1′ in the received light, andblue color component B1′ in the received light to third surface 541R(541L) in reflective unit 540R (540L). Coated surface of 541R (541L)reflects all wavelengths received from coated first surface 731R (731L)onto image capture sensor 715R (715L).

Each of image capture sensors 710R (710L) and 715R (715L) is a colorsensor with a color filter array in this aspect. The color filter arrayis a Bayer RGGB color filter array. Thus, the two Bayer pattern imagecapture sensors are looking through the same optics at the same scene.Here, a Bayer pattern image capture sensor is a single chip sensor, orpart of a single chip, that includes a Bayer color filter array. Asnoted above, coplanar image capture sensors 710R (710L) and 715R (715L)have a common front end optical structure and substantially the sameoptical path length to each sensor.

When prism assembly 730R (730L) and reflective unit 540R (540L) arearranged so that the color images captured by both Bayer pattern imagecapture sensors 710R (710L) and 715R (715L) are the same scene, thecolor image captured by image capture sensor 710R (710L) has the samefull resolution as the color image captured by image capture sensor 715R(715L). Thus, each point in space in the scene is represented by twopixels with each of the two pixels having a different three colorcomponent vector. Thus, sensor assembly 720R (720L) has acquired a fullresolution image with six primary color components—three from the firstcaptured image and three from the second captured image—per pixel withno loss of light due to the notch filters.

The color component vector for each output pixel to stereoscopic display251 from dual image enhancement module 240R (240L) in central controller260 is derived from the six primary color components for this pixel.This permits driving a stereoscopic display with for example, a fourelement color vector for each pixel without any loss in resolution. Dualimage enhancement module 240R (240L) samples the two images and appliesa color correction matrix to generate the pixel vector required bystereoscopic display 251.

In this aspect, the coatings—the notch filter—on coated first surface731r (731L) provide the flexibility to pick the color separation. Thecolor separation of coated first surface 731R (731L) is, of course,multiplied by the pixel RGGB pattern. Thus, if coated first surface 731R(731L) is separating out yellow for example, it is expected that thiswould excite both green and red pixels and this would then be recoveredin the color correction matrix process of the imaging pipeline.Similarly, the coated first surface can be configured to pass bluewavelengths in a fluorescence spectrum, and reflect all otherwavelengths of blue, for example.

As described above, the color correction matrix acts upon a six elementvector per pixel (a three color component vector of a correspondingpixel in a first image captured by image capture senor 710R (710L) plusa three color component vector of a corresponding pixel from a secondimage captured by image capture sensor 715R (715L)) and produces anoutput color component vector with the number of color components in thedisplay—for a “normal” LCD, this is the red, green blue (RGB)colorcomponents, and for a Sharp five color display, this is the red, redcomplement, yellow, green, blue (RR′YGB) color components.

More specifically, color correction module 241R (241L) in dual imageenhancement module 240R (240L) of central controller 260 is coupled tothe first and second image capture sensors 710R (710L) and 715R (715L).Camera control unit 230R (230L) demosaics the first image captured bythe first image capture sensor 710R (710L) and generates an N-elementcolor vector for each pixel N typically is three. Camera control unit230R (230L) similarly &mosaics the second image captured by the secondimage capture sensor 715R (715L) into pixels, where each pixel ispresented by a second N-element color vector. Dual image enhancementmodule 240R (240L) then processes the two N-element color vector as a 2N(typically six) color vector through color correction color correctionmodule 241R (241L) to create an output color vector for each pixel withthe desired number of color components. The output color vector for atypical liquid crystal display would have three color components. For aSharp display with five primaries, the output color vector would be afive element color vector. A specific color correction matrix would bechosen based on optimization to achieve the best color performance bysome metric. The metric could be, for example, to match the color seenby a human observer with Xenon illumination for some set of tissuetypes.

Enhanced Color Performance—White Broadband Illumination and EnhancedDepth of Field

In a third aspect, illuminator 710C (FIG. 7C) is coupled to illuminationchannel 705 (FIG. 7A). Illuminator 710C is the same as that describedabove and so that description is not repeated here. Thus, the scene isilluminated by broadband white light from illuminator 710C.

102461 As noted above, the reflected light from tissue 703 passesthrough lens element 704R (704L) and stop 570R (570L). In this aspect,lens element 704R (704L) is different from the lens element consideredabove for the other aspects related to FIG. 7A. In the other aspect,lens element 704R (704L) is designed to correct for longitudinal coloraberrations so that the various color components focus on about the sameplane. In this aspect, lens element 704R (704L) does not correct forlongitudinal color aberrations. Instead, lens element 704R (704L) isdesigned to focus different wavelengths of light at different distancesfrom the lens, i.e., is designed to have a significant and controlledamount of longitudinal color. For example, lens groups 482 and 484 inlens assembly 401B (FIG. 4B) are designed to have a significant andcontrolled amount of longitudinal color. Such lens groups are known tothose knowledgeable in the field and so is not considered in furtherdetail herein.

In this aspect, coated first surface of 731R (731L) of prism assembly730R (730L) includes filter 790F (FIG. 7F) that was previouslydescribed. To avoid repetition., the above description of filter 790F isnot repeated here.

Hence, coated first surface of 731R (731L) reflects red color componentR1 in the received light, green color component G1 in the receivedlight, and blue color component B1 in the received light to secondsurface 732R (732L). Coated surface of 732R (732L) reflects allwavelengths received from coated first surface 731R (731L) onto imagecapture sensor 710R (710L).

Coated first surface of 731R (731L) transmits red color component R1′ inthe received light, green color component G1′ in the received light, andblue color component B1′ in the received light to third surface 541R(541L) in reflective unit 540R (540L). Coated surface of 541R (541L)reflects all wavelengths received from coated first surface 731R (731L)onto image capture sensor 715R (715L).

Again, each of image capture sensors 710R (710L) and 715R (715L) is acolor sensor with a color filter array in this aspect. The color filterarray is a Bayer RGGB color filter array. Thus, the two Bayer patternimage capture sensors are looking through the same optics at the samescene. Here, a Bayer pattern image capture sensor is a single chipsensor, or part of a single chip, that includes a Bayer color filterarray. As noted above, coplanar image capture sensors 710R (710L) and715R (715L) have a common front end optical structure and substantiallythe same optical path length to each sensor.

When prism assembly 730R (730L) and reflective unit 540R (540L) arearranged so that the color images captured by both Bayer pattern imagecapture sensors 710R (710L) and 715R (715L) are the same scene, thecolor image captured by image capture sensor 710R (710L) has the samefull resolution as the color image captured by image capture sensor 715R(715L). Thus, each point in space in the scene is represented by twopixels with each of the two pixels having a different three elementvector. Thus, sensor assembly 720R (720L) has acquired a full resolutionimage with six primary color components—three from the first capturedimage and three from the second captured image—for each point in spacewith no loss of light due to the notch filters. However, due to thelongitudinal color, the captured images are blurred different amounts ineach of the six color primaries acquired.

Hence, in this aspect, dual image enhancement module 240R (240L)includes a digital filter kernel for each of the first and secondimages. The digital filter kernel processes the captured image based onthe known longitudinal color aberrations to enhance the sharpness andfocus of the image, i.e., to generate third and fourth images. The thirdand fourth images are demosaiced. The resulting demosaiced third imageand fourth image can each be focused at a range of distances from thelens assembly and so as to provide an image having a greater depth offield than from a traditional lens design that brought all the colorcomponents into focus. Digital filter kernels are known to thoseknowledgeable in the field. This approach works well for surfaces withsmooth reflectance curves in wavelength; this is the case for mosttissue.

The vector for each output pixel to stereoscopic display 251 from dualimage enhancement module 240R (240L) in central controller 260 isderived from the six primary color components for pixels in thedemosaiced third and fourth images. Dual image enhancement module 240R(240L) samples the third and fourth images created by the digital filterkernel and applies a color correction matrix to generate the pixelvector required by stereoscopic display 251. The resulting combinedimage has a depth of field that is potentially three times greater thanan image obtained from the conventional lens system.

Enhanced Color Performance—Three Color Component Illumination andFluorescence

In a fourth aspect, an illuminator 710D (FIG. 7D) is coupled toillumination channel 705 (FIG. 7A). Illuminator 710D includes threelaser illumination sources 710D1 to 710D3 that generate a red colorcomponent R1, a green color component G1, and a blue color component B1,respectively. Sometimes herein, three laser illumination sources 710D1to 710D3 are referred to as three laser illumination modules. Theconfiguration of an illuminator with multiple different illuminationsources, a mirror for source 710D1 and dichroic mirrors for sources710D2 and 710D3 is known. With laser illumination sources, thewavelengths of red color component R1, a green color component GI, andblue color component B1 are typically 2 to 5 nanometer (nm) wide insteadof about 60 nm per color component from a conventional light emittingdiode based illuminator.

The light from the red, green and blue lasers reflects off tissue 703.The intensities of the reflected color component R1, color component G1,and color component B1 are modulated by the reflectance function of thetissue. In addition, to the three reflected color components fromilluminator 710D, tissue 703 may emit other light. For example, thelight from one of the lasers may excite fluorescence. See for examplespectrum 795G (FIG. 7G). The three peaks are light reflected by tissue703 from the red, green and blue lasers of illuminator 710D. Therelatively smaller and broader peak in the red region of the spectrum isfluorescence. Spectrum 795G is presented to aid in visualization of thelight that enters endoscope 702. and is not representative of any actualdata.

Sensor assembly 720R (720L) enables capture of a color image by firstimage capture sensor 710R (710L) and enables all the light notassociated with the color image to be redirected to second image capturesensor 715L, (715). Thus, as explained more completely below, sensorassembly 720R (720L) also enables imaging fluorescence in real time atnearly any wavelength in the visible or near infrared, and enablesimaging natural tissue fluorescence.

Fluorescent markers may be stimulated by a narrow band source like alaser 710E4 in illuminator 710E that emits a wavelength in the nearinfrared spectrum. The fluorescent emission spectrum, however, is arange of wavelengths. The shape of the fluorescent emission spectrum isnot significantly dependent on the excitation wavelength in a range ofexcitation wavelengths for the fluorescence.

Thus, in one aspect, fluorescence is triggered by light from a lasermodule 710E4 in illuminator 710E. As an example, antibody agents, whichwere obtained from Mendarex, Inc., were excited using a 525 nm laser. Inanother aspect, illuminator 710D includes one or more light modules inaddition to the three laser modules that generate red color component R,green color component G1, and blue color component B1. The number andtype of additional light modules are selected based on the fluorescenceexcitation wavelengths for one or more fluorescences of interest.

The particular fluorescence excitation source selected for combinationlight source 210 depends on the fluorophore or fluorophores used.Excitation and emission maxima of various FDA approved fluorescent dyesused in vivo are presented in Table 1.

TABLE 1 Excitation maxima Emission maxima Fluorescent Dye (nm) (nm)Fluorescein 494 521 Indocyanine Green 810 830 Indigo Carmine ® 436 inalkaline 528 in alkaline solution solution Methylene Blue 664 682 IndigoCarmine ® is a U.S. registered trademark of Akorn. Inc. of Lake Forrest,Ill, USA.

Table 2 presents examples of common protein fluorophores used inbiological systems.

TABLE 2 Fluorescent proteins/ Excitation maxima Emission maximaFluorophore (nm) (nm) GFP 489 508 YFP 514 527 DsRed (RFP) 558 583 FITC 494**  518** Texas red  595**  615** Cy5  650**  670** Alexa Fluor 568 578**  603** Alexa Fluor 647  650**  668** Hoechst 33258 346 460TOPRO-3 642 661 **Approximate excitation and fluorescence emissionmaxima for conjugates.

Those knowledgeable in the field understand that a fluorophore can bebound to an agent that in turn binds to a particular tissue of thepatient. When a particular fluorophore is selected, if one of the threevisible color component laser modules in illuminator 710D does notprovide the necessary excitation wavelength, another laser module 710E4can be added to illuminator 710D to obtain illuminator 710E thatprovides light with the excitation maxima wavelength for thatfluorophore. Thus, given the fluorophore or fluorophores of interest andthe number of different fluorophores used, appropriate light sources canbe included in illuminator 710D to provide illumination of a color imageof the scene and to excite fluorescence.

The above examples in Tables 1 and 2 are illustrative only and are notintended to limit this aspect to the particular examples presented. Inview of this disclosure, an alternate imaging characteristic of thetissue can be selected and then an appropriate light source can beselected based upon the fluorescence being utilized.

In this aspect, coated first surface of 731R (731L) of prism assembly730R (730L), i.e., the beam splitter, includes a filter 790H (FIG. 7H)that includes a plurality of notch filters. A first notch filter 791Hreflects red color component R1 in the received light and transmitsother wavelengths of red color component R in the received light. Asecond notch filter 792H reflects green color component in the receivedlight and transmits other wavelengths of green color component G in thereceived light. A third notch filter 793H reflects blue color componentB1 in the received light and transmits other wavelengths of blue colorcomponent B in the received light. The use of three notch filters is forease of discussion and is not intended to be limiting to three separatefilters. The three separate notch filters can be viewed as a singlemulti-notch filter with the reflect, transmit properties described here.

Hence, coated first surface of 731R. (731L) reflects red color componentR1 in the received light, green color component G1 in the receivedlight, and blue color component B1 in the received light to secondsurface 732R (732L). Second surface of 732R (732L) reflects all.wavelengths received from coated surface 731R (731L) onto image capturesensor 710R (710L).

Coated first surface of 731R (731L) transmits red light other than redcolor component R1 in the received light, green light other than greencolor component G1 in the received light, and blue light other than bluecolor component B1 in the received light to third surface 541R (541L) inreflective unit 540R (540L). Third surface of 541R (541L) reflects allwavelengths received from coated surface 731R (731L) onto image capturesensor 715R (715L).

Thus, image capture sensor 710R (710L) acquires an RGB image of thelaser illuminated scene. This is a color image. All the remaining lightis reflected onto image capture sensor 715R (715L). This light must befrom a source other than the illumination. For example, if there isnatural tissue fluorescence, the fluorescence is captured by imagecapture sensor 715R (715L). If image capture sensor 715R (715L) is acolor image capture sensor, the fluorescence can be of any color and thefluorescence is imaged in color on image capture sensor 715R (715L).There may be slight losses in the fluorescence due to the notch filterson coated first surface of 731R (731L). In this aspect, image capturesensor 710R (710L) is a color sensor with a color filter array. Thecolor filter array is a Bayer RGGB color filter array.

The type of image capture sensor selected for image capture sensor 715R(715L) depends on the characteristics of the other light that iscaptured by image capture sensor 715R (715L). The gain and otherparameters of image capture sensor 715R (715L) may be independent ofimage capture sensor 710R (710L) so that image capture sensor 715R(715L) acquires the best image possible.

If a single fluorescent spectrum is of interest, or if distinguishingbetween different fluorescent spectra is not of interest, image capturesensor 715R (715L) can be a monochrome sensor. Alternatively, if morethan one color fluorescent spectrum is of interest, e.g., a fluorescentspectrum in the red wavelengths and a fluorescent spectrum in the greenwavelengths, image capture sensor 715R. (715L) can be a color imagecapture sensor with a Bayer color filter array.

With a monochrome image capture sensor, there is a benefit in terms ofresolution and light gathering. The color image capture sensor givesmore information about the wavelength or wavelengths of the lightreceived.

Irrespective of the implementation of image capture sensor 715R (715L),the two image capture sensors are looking through the same optics at thesame scene. As noted above, coplanar image capture sensors 710R (710L)and 715R (715L) have a common front end optical structure andsubstantially the same optical path length to each sensor.

Any small amount of light, which goes to image capture sensor 710R(710L) and is not reflected laser illumination, is small relative to thereflected laser illumination. Assuming broad spectral responses and goodregistration of the images captured by image capture sensors 710R (710L)and 715R (715L), dual image enhancement module 240R (240L) could correctthe image captured by image capture sensors 710R (710L) using the imagecaptured by image capture sensor 715R (715L).

If fluorescence is excited by an excitation wavelength from anillumination module different from the illumination modules inilluminator 710D that generate light having wavelengths R1, G1, B1, anexcitation blocking notch filter is required in the image capture unitimaging path, e.g., between stop 570R (570L) and sensor assembly 720L(720R). Irrespective of the excitation wavelength of the fluorescence,the fluorescent light can be in the visible spectrum and be imaged ontoimage capture sensor 715R (715L). Multiple fluorophores could be imagedsimultaneously if desired and the emission spectra excite differentcolors on image capture sensor 715R (715L).

The configuration of dual enhancement module 240R (240L) depends on thewhether second image capture sensor 715R (715L) is a color image capturesensor or a monochrome image capture sensor. If both image capturesensors are color sensor, CCU 230R (230L) demosaics the two capturedimages and controller 260 generates an N-element color component vectorfor each pixel in the output image as described above. Three of thecolor components in the N-element color component vector represent acolor image and the remaining color components represents fluorescence.the second image capture sensor is monochrome, the demosaicing of thesecond captured image is unnecessary.

Thus, irrespective of the implementation of second image capture sensor715R (715L), controller 260 is configured to generate at least a fourelement color component vector for a pixel in an output image from acolor component vector for a corresponding pixel in the first image anda color component vector for a corresponding pixel in the second image.Three of the elements in the four element color component vector are fora visible color image, and a fourth element in the four element colorcomponent vector is for a fluorescent image.

In the aspect of FIG. 7A and each of the examples described above withrespect to FIG. 7A, the image capture sensors in a stereoscopic channelare coplanar. This is illustrative only and is not intended to belimiting. For example, a first of the image capture sensors could have atop sensor surface in a first plane, and a second of the image capturesensors could have a top sensor surface in a second plane. The first andsecond planes are parallel to each other and are separated by a knowndistance (Sec FIG. 9). The spacing between the beam splitter and thereflective surface in the reflective unit is adjusted to compensate forthe known distance so that the first optical path length to the firstimage capture sensor is about equal to the second optical path length tothe second image capture sensor. Thus, each of the aspects of FIG. 7Adescribed above are directly applicable to this configuration as the twooptical path lengths remain about equal.

Extended Depth of Field

In a surgical endoscope, one would like to provide as sharp and asbright an image as possible to the surgeon. This requires a lens designwith the largest possible aperture. A larger aperture forms a sharperimage (within the limitations of the sensor) and allows more lightthrough leading to a brighter image (better signal to noise ratio).However, there is a trade-off in that a larger aperture leads to ashallower depth of field. The surgeon would prefer an image with agreater depth of field to avoid having to control the focus of theimaging system as the surgeon looks at things at different distancesaway from the camera. Thus, the surgeon would like an image with thesharpness and brightness of a large aperture and much improved depth offield. To achieve this goal, each of image capture units 825L and 825R(FIG. 8A) capture images of tissue 803 with different focus. Dual imageenhancement modules 240R, 240L (FIG. 2) process the captured images andprovide the surgeon with depth of field capabilities not previouslyavailable.

For example, in one aspect, a first image 881 (FIG. 8B) captured bysensor assembly 820R (820L) has a first focus. A second image 882 (FIG.8C) captured by sensor assembly 820R (820L) has a second focus. In image881, the features in the center of the image are in focus as representedby the dark distinct lines. The features in the left and right sides ofimage 881 are further from image capture unit 825R (825L) and are out offocus as represented by the cross-hatching. In image 882, the featuresin the center of the image are out of focus as represented by thecross-hatching. The features in the left and right sides of image 882are in focus as represented by the dark distinct lines.

As explained more completely below, the in focus portions of images 881and 882 are used to generate a composite in focus image that is sent tostereoscopic display 251 (FIG. 8D). This provides an image containingbetter depth of field than is available in either of images 881 and 882.Another benefit of this approach is that information from images 881 and882 along with a sharpness metric are used to generate an image from aperspective of virtual cameras that are sent to stereoscopic display 251for display. This allows the surgeon to view tissue 803 from slightlydifferent perspectives without physically moving endoscope 802.

FIG. 8A is a schematic illustration of a distal end of a stereoscopicendoscope 802 with image capture units 825L, 825R and an illuminationchannel 805 that provides light from an illuminator. As indicated byarrow 835, the distal direction is towards tissue 803 and the proximaldirection is away from tissue 803.

Each image capture unit 825R, 825L includes a lens assembly 801R, 801Land a sensor assembly 820R, 820L. Sensor assembly 820R, 820L ispositioned to receive light that passes through lens assembly 801R,801L. Each sensor assembly 820R, 820L includes a prism assembly 830R,830L, a reflective unit 840R, 840L, and coplanar image capture sensors(810R, 815R), (810L, 815L), in one aspect. However, the configurationillustrated in FIG. 9 could also be used in the aspects described usingthe configuration of FIG. 8A. Stereoscopic endoscope 802 is sometimesreferred to as endoscope 802.

In the following description, the optical paths in the right channel ofstereoscopic endoscope 802 are described. The optical paths through theleft channel of stereoscopic endoscope 802 are equivalent those in theright channel due to the symmetry of endoscope 802, and so in thefollowing description, the left channel reference numeral is includedwithin parentheses following the description of an element in the rightchannel. This indicates that the description is also applicable to thecorresponding element in the left channel. Also, the imaging processingthat is described is performed in real time so that a continuous videosequence is provided to the display.

The light from tissue 803 passes through lens element 804R (804L) andstop 870R (870L) in lens assembly 801R (801L), The elements in lensassembly 401B (FIG. 4B) are an example of lens element 804R (804L). Thelight that passes through lens assembly 801R (801L) is received by imagecapture unit 825L (825R) and enters prism assembly 830R. Prism assembly830R (830L) includes a beam splitter 831R (831L) that reflects a firstpercentage of the received light. Beam splitter 831R (831L) passes asecond percentage of the received light through beam splitter 831R(831L) to reflective unit 840R (840L).

In one aspect, beam splitter 831R (831L) is implemented as a coatedfirst surface 831R (831L). The light reflected by coated first surface831R (831L) is received by a second surface 832R (832L) that in turndirects, e.g., reflects, the light onto first image capture sensor 810R(810L), Surface 832R (832L) can be either a coated surface or a totalinternal reflection surface. The light transmitted through coated firstsurface 831R (831L) is received by a third surface 541R (541L) ofreflective unit 540R (540L) that in turn directs, e.g., reflects, thelight onto second image capture sensor 815R (815L).

Second surface 832R (832L) is positioned so that no light other than thelight reflected by coated first surface 831R (831L) hits second surface832R (832L). In one aspect, the angle of incidence of the light tocoated surface 831R (831L) is less than forty-five degrees.

In one aspect, prism assembly 830R (830L) and reflective unit 840R(840L) are included in a prismatic structure with a pentaprism thatincludes prism assembly 830R (830L). The prismatic structure in thisaspect is equivalent to prismatic structure 460 (FIG. 4B) with buriedcoated surface 431B that splits the received light into the first andsecond percentages. Thus, the description of prismatic structure 460 isapplicable to the prismatic structure used in this aspect.

In one aspect, the first and second percentages are about equal. Inanother aspect, the first and second percentages are different. Thus,prism assembly 830R (830L) includes a beam splitter 831R (831L) thatseparates the received light into the two portions—(i) a firstpercentage of the received light sometimes referred to as a firstportion, and (ii) a second percentage of the received light, sometimesreferred to as a second portion.

Image capture sensors 810R (810L) and 815R (815L) are coplanar. In oneaspect, a first optical path length from stop 870R (870L) to coatedfirst surface 831R (831L) to second surface 832R (832L) to image capturesensor 810R (810L) is a first length. A second optical path length fromstop 870R (870L) through coated first surface 831R (831L) to a thirdsurface 841R (841L) in assembly 840R (840L) to image capture sensor 815R(815L) is a second length. The first and second lengths are different,i.e., are not about equal. Again, the definition of the optical pathlengths as starting at stop 870R (870L) is illustrative and is notintended to be limiting. The unequal optical path lengths could also bevariously defined, such as with respect to a distal face of prismassembly 830R (830L) through which the received light enters prismassembly 830R (830L), with respect to a first element in lens assembly801R (801L), or with respect to coated first surface 831R (831L). Inanother aspect, the different optical path lengths can be achieved withglass plates on the two image sensor areas with different indexes ofrefraction or other similar means.

Thus, coplanar image capture sensors 810R (810L) and 815R (815L) have acommon optical path length through the front end optical structure inlens assembly 801R (801L) and different optical path lengths to eachsensor in sensor assembly 820R (820L). First image capture sensor 810R(810L) captures an image from the first portion of the light received bysensor assembly 820R (820L). Second image capture sensor 815R (815L)captures an image from the second portion of the light received bysensor assembly 820R (820L).

In one aspect, each of image capture sensors 810R (810L) and 815R (815L)is a color sensor with a color filter array. In another aspect, thecolor filter array is removed from one of the color sensors and thesensor functions as a monochrome sensor. When a monochrome sensor isused, the sharpness from the image captured by the monochrome sensor istransferred to the color image if the focus of the two images is not toodifferent, e.g., the two images are focused in the vicinity of the pointwhere sharpness curves 885, 886 of the two images intersect (See FIG.8E).

In one aspect, coated first surface 831R (831L) of prism assembly 830R(830L) is configured to reflect and to transmit about equal portions ofthe light received by prism assembly 830R (830L), i.e., the first andsecond percentages are about equal. Thus, beam splitter 831R, in thisaspect, is the balanced beam splitter described above. In this example,each of image capture sensors 810R (810L) and 815R (815L) is a colorsensor with a color filter array. The color filter array is a Bayercolor filter array. Thus, the two Bayer pattern image capture sensors810R (810L) and 815R (815L) are looking through the same optics at thesame scene. Here, a Bayer pattern image capture sensor is a single chipsensor, or part of a single chip, that includes a Bayer color filterarray. As noted above, coplanar image capture sensors 810R (810L) and815R (815L) have a common front end optical structure, but differentoptical path lengths to each sensor.

The light going to image capture sensor 810R (810L) is captured as afirst image focused at a first object distance DA from image captureunit 825R (825L). See for example FIG. 8B. The light going to imagecapture sensor 815R (815L) takes a slightly longer path to arrivessensor at 815R (815L) and is captured as a second image focused at asecond object distance DB from the image capture unit. See for example,FIG. 8C. Distance DB is larger than distance DA. The difference in thepath lengths is based on the front end optical design, e.g., focallength of lens element 804R (804L), and the distance for the best focusof each of the two captured images.

The first and second images are aligned and taken through the same lenselement 804R (804L). The focal plane of the two images is different. Thedepth of field of the two images is also different. The image focused atdistance DA, e.g., image 881, has a short depth of field whereas theimage focused at distance DB, e.g., image 882, has a broader depth offield.

As noted above, the surgeon can use user interface 262 to cause dualimage enhancement module 240R (240L) to generate, from the two capturedimages (FIGS. 8B and 8C), an in focus image (FIG. 8D) with an enhancedfield of view Alternatively, the surgeon can show the two images on twoplanes of a display so the accommodation in surgeons control consoledisplay 251, sometimes called stereoscopic display 251, matches thesurgical accommodation at the surgical site. Finally, the surgeon canswitch between the first image with one focus and the second image witha second focus.

In one aspect, dual image enhancement module 240R (240L) generates an infocus extended depth of field image from the two images captured byimage capture sensors 810R (810L) and 815R (815L). In a first aspect, afirst sharpness metric is generated for a group of pixels, i.e., a tile,in the first image and a second sharpness metric is generated for acorresponding group of pixels in the second image. The two sharpnessmetrics are compared to determine which sharpness metric is larger. Thegroup of pixels (or a selected pixel or pixels within the tile) with thelargest sharpness metric is selected for the blended image. Dual imageenhancement module 240R (240L) steps through the two captured imagesgroup by group and selects one of the two groups for inclusion in theblended image that is in focus over a greater field of view than the twocaptured images.

FIG. 8E is an illustration of a sharpness profile 885 for imagedcaptured by image capture sensor 810R (810L) and a sharpness profile 886for the images captured by image capture sensor 815R (815L). Thesharpness profile is a curve of the sharpness of an image captured by animage capture sensor versus the distance from the distal end of theendoscope. In one aspect, sharpness profiles 885 and 886 are empiricallygenerated based on a calibration performed using endoscope 802 and aplurality of images taken of known objects at different distances fromthe distal end of endoscope 802.

In one aspect, profiles 885 and 886 are stored as look-up tables in amemory. The stored sharpness profiles are used by dual image enhancementmodule 240R (240L) to determine whether to display a first imagecaptured by image capture sensor 810R (810L) or to display a secondimage captured by image capture sensor 815R (815L) as endoscope 802 ismoved without moving the optics in image capture unit 825R (825L). Dualimage enhancement module 240R (240L) generates a sharpness metric, usinga conventional technique know to those knowledgeable in the field, foreach of the first and second images. Dual image enhancement module 240R(240L) uses sharpness profiles 885 and 886 and the sharpness metrics todetermine a distance between the distal end of endoscope 802 and thetissue for each of the images.

For example, if the first image has a sharpness metric of 0.5, distanceDA to tissue 803 is either about forty-two millimeters (mm) or abouttwenty seven millimeters (mm) based on sharpness profile 885. If secondimage has a sharpness metric of about 0.43, distance DB to tissue 803 isabout forty-two millimeters (mm) based on sharpness profile 886.Alternatively if second image has a sharpness metric of about 0.08, thenthe tissue distance is about twenty seven millimeters (mm). Note thatsharpness profile 885 is ambiguous as to tissue depth based on thesharpness metric; the tissue depth is disambiguated by second curve 886.As both images are obtained at substantially the same time, thisdisambiguation is not skewed by temporal effects. Dual image enhancementmodule 240R (240L) uses the relative sharpness metric between the twoimages and the a-priori calibration of the curves to determine which ofthe two images to display as endoscope 802 is moved and the distancefrom the image capture unit to the tissue varies. The position of thedistal end of the endoscope is known by central. controller 260.

If endoscope 802 is moved closer to tissue 803, the first captured isimage is sent to display 251 by dual image enhancement module 240R(240L). If endoscope 802 is moved away from tissue 803, dual imageenhancement module 240R (240L) compares the distance from tissue 803 ofthe distal end of endoscope 802 with the distance at which sharpnessprofiles 885 and 886 intersect, referred to as the profile intersectiondistance. If the distance from tissue 803 of the distal end of endoscope802 is smaller than the profile intersection distance, the firstcaptured image is transmitted to display 251. If the distance fromtissue 803 of the distal end of endoscope 802 is larger than the profileintersection distance, the second captured image is transmitted todisplay 251. Thus, for the sharpness metrics in the previous paragraph,when the distal end of end of endoscope 802 is less than 45 mm fromtissue 803, the first image would be presented to the surgeon. When thedistal end of end of endoscope 802 is more than 45 mm from tissue 803,the second image would be presented to the surgeon. Thus, as endoscope802 is moved, the surgeon is presented with an appropriate image withouthaving to move the optics in image capture unit 825R (825L).

Additionally, the determination of which of the two images to present tothe surgeon may be determined solely on the basis of the relativesharpness metric; the calculation of tissue depth is not necessary. Theimage pair presented in the left and right eves of the stereo viewerwould be kept consistent and the sharpness metric comparison may use thedata from the pair of images captured for the left eye and the pair ofimages captured for the right eye during the determination to use thefirst or second image. In general, if switching focus of the entireimage presented, the first image would be chosen for both the right andleft eyes or the second image would be chosen for the right and lefteyes.

In another aspect, the captured two images and sharpness profiles 885,886 are used to create a depth map. Sharpness profiles 885 is dividedinto sharpness profile 886 to generate a channel sharpness profile of asharpness metric versus distance e.g., the value of profile 886 at agiven distance is divided by the value of profile 885 at that givendistance to obtain the sharpness metric for the given distance. Thissharpness metric may be calculated through the use of sliding tiles tocreate a depth map for the entire image. Naturally, this information maybe combined with more traditional stereo matching algorithms to generatea more well conditioned depth map.

In one aspect, sharpness profiles 885 and 886 are empirically generatedbased on a calibration performed using endoscope 802 and a plurality ofimages taken of known objects at different distances from the distal endof endoscope 802. Sharpness profiles are obtained for both image captureunit 825R and for image capture unit 825L. A channel sharpness profileis generated for each of the left and right channels. In one aspect, theleft and right channel sharpness profiles are saved as a look-up tablein a memory 899 (FIG. 8F).

Recall that, there is essentially a one to one correspondence betweenthe pixels in the first captured image and the pixels in the secondcaptured image. Establishing the one to one correspondence, in someinstances, may require calibration and slight image warping or softwarebased registration between the images. Dual image enhancement module240R (240L) generates a sharpness metric, using a conventional techniqueknow to those knowledgeable in the field, for tiles or groups aroundcorresponding pixels in each of the first and second images. Thesharpness metric for the pixel, or a group of pixels about the pixel, inthe first captured image is divided into the sharpness metric for thecorresponding pixel, or corresponding group of pixels about thecorresponding pixel, in the second capture image to generate a channelsharpness metric for the pixels. The channel sharpness metric for thepixels is used to locate a point on the channel sharpness profile. Thedistance corresponding to that point on the channel sharpness profile isthe depth associated with the (X,Y) position of the pixel. Performingthis process for each pixel in the captured images generates a depth mapfor the captured images.

In one aspect, the depth map is used to generate view of a scene oftissue 803 from a perspective different from the perspective ofendoscope 802, i.e., from a perspective of a virtual camera view point.Central controller 260 uses a three-dimensional virtualization module890 (FIG. 8F) to generate a stereoscopic image pair for the virtualcamera view points.

Three-dimensional virtualization module 890 uses the first and secondimages captured in the right stereoscopic channel of endoscope 802, andgenerates a first channel depth map for the scene viewed by the rightchannel. Similarly, three-dimensional virtualization module 890 uses thefirst and second images captured in the left stereoscopic channel ofendoscope 802, and generates a second channel depth map for the sceneviewed by the left channel. Additional information from stereo matchingin the images may also be used to improve the stability and accuracy ofthe combined depth map. A three-dimensional surface of the scene isgenerated using the first and second channel depth maps and the bestfocused portions of the four captured images are projected on andtextured onto the three-dimensional surface to create a texturedthree--dimensional image surface.

When a surgeon inputs a view point using user interface 262, i.e., avirtual camera view point, dual image enhancement module 240R uses thechannel depth map for the right channel and the texturedthree-dimensional image surface to generate an image for the rightchannel from the virtual camera view point. Similarly, dual imageenhancement module 240L. uses the channel depth map for the left channeland the textured three-dimensional image surface to generate an imagefor the left channel from the virtual camera view point. The two imagesare sent to stereoscopic display 251 and the surgeon is presented avirtual three-dimensional image without moving endoscope 802 to thevirtual camera view point.

More specifically, a 3-D virtualization module 895 is executed for eachchannel. This can be done either in parallel or sequentially dependingon the memory and processing power available in central controller 260.

In RETRIEVE IMAGES process 891 for the right channel, first and secondimages captured in image capture sensors 810R and 815R are retrieved andprovided to CREATE DEPTH MAP process 892, sometimes called process 892.Process 892 performs the same process repetitively for groups of pixels,e.g., tiles, in the two images.

In process 892 for a pixel (X,Y), a group of pixels including pixel(X,Y) are selected from each of the first and second images in the rightchannel. Here, X is the x-coordinate of the pixel and Y is they-coordinate of the pixel. For example, the group of pixels could be afive pixel by five pixel block, with pixel (X,Y) at the center of theblock. A first sharpness metric for the block of pixels from the firstimage is generated and a second sharpness metric for the same block ofpixels from the second image is generated. The second sharpness metricis divided by the first sharpness metric to generate a channel sharpnessmetric. A distance Z corresponding to the channel sharpness metric isobtained using right channel sharpness profile in memory 899. Distance Zis assigned to pixel (X,Y). Thus, in the right channel depth map thepixel is represented as pixel (X, Y, Z,_(xy).)

Next this process is repeated for pixel (X+1, Y), and the pixel in thedepth map is pixel (X+1, Y, Z_((x+1)y)). Thus, some of the pixels usedin determining the common block sharpness metric for pixel (X, Y) arealso used in determining the common block sharpness metric for pixel(X+1, Y). The process is repeated for each pixel in a row of pixels andthen the Y index is incremented and the process repeated for each pixelin the next row pixels. For pixels in a boundary region of the scenethat are not surrounded by a complete block of pixels, symmetry isassumed and the values for pixels available in the block are used togenerate values for the missing pixels in one aspect.

When all the rows of pixels are processed, a right channel depth map hasbeen created with each pixel in the scene viewed by the right channel.Each pixel has a value in the x-dimension, a value in the y-dimension,and a depth in the z-dimension. The right channel depth map is stored ina memory 895, in one aspect.

The depths in the right channel depth map are averages values that are afunction of the size of the group of pixels. A large group of pixelsreduces noise, but as the group becomes larger detail in the z-dimensionis lost due to the averaging over the group. Thus, a size for the groupis empirically determined to reduce the noise to an acceptable levelwhile maintaining the desired specificity in the depth values, Also, theplacement of the pixel for which the depth is being generated at thecenter of block is illustrative only and is not intended to be limitingto this specific placement.

This same process is performed for the left channel by create depth mapprocess 892. When the processing for both the right and left channels iscompleted, memory 895 includes a right channel depth map and a leftchannel depth map. The left and right channel depth maps provide a depthmapping of the scene for the view point of endoscope 802.

GENERATE 3-D SURFACE process 893 in central controller 260 uses the leftand right channel depth maps to generate a three dimensional surface ofthe scene. The use of two depths maps to generate a three dimensionaldepth map surface is known to those knowledgeable in the field and so isnot considered further herein.

After generating, the three dimensional surface of the scene, PROJECTAND TEXTURE MAP IMAGES process 894 projects and texture maps thesharpest portions of the two acquired images in the right channel andthe sharpest portions of the two acquired images in the left channelback on the three dimensional depth map surface to generate a texturedthree-dimensional image surface, sometimes referred to as a texturedimage surface. Projecting and texturing using a plurality of capturedimages and a three dimensional surface to generate a textured imagesurface is known to those knowledgeable in the field and so is notconsidered further herein.

With the textured image surface and the two channel depth maps, imagesfor a stereoscopic image pair from a virtual camera view point can begenerated. In this aspect, user interface 262 permits a surgeon tospecify a virtual camera view point from which the surgeon wishes toview the current scene, e.g., a different perspective from which thesurgeon wishes to view the current scene.

Hence, RECEIVE VIRTUAL CAMERA VIEW POINT process 897 receives thevirtual camera view point from user interface 262 and provides thevirtual camera view point to GENERATE IMAGES process 896. GENERATEIMAGES process 896 uses the right channel depth map from DEPTH MAPS 895,the textured image surface, and the virtual camera view point togenerate a virtual right channel image from the virtual camera viewpoint. Similarly, GENERATE IMAGES process 896 uses the left channeldepth map from .DEPTH MAPS 895, the textured image surface, and thevirtual camera view point to generate a virtual left channel image fromthe virtual camera view point. Generating a virtual image using a depthmap and a textured image surface for a virtual camera view point is knowto those knowledgeable in the field and so is not considered in furtherdetail.

Upon generation of the left and right channel virtual images for thevirtual camera view point, TRANSMIT IMAGES process 898 sends the twovirtual images to stereoscopic display 251 for display to the surgeon.Thus, without moving endoscope 802, the surgeon is able to view thescene from a different view point, i.e., a virtual camera view point.Naturally, viewing the scene from a view point wildly different thanthat of endoscope 802 does not necessarily lead to a high quality image.However, images for virtual camera points near the endoscope tip are ofreasonable quality.

This approach may also be used for generating the images for otherparticipants in an operating room. For example, the operating room staffnear the patient likely views the surgical procedure on a non-stereodisplay unit, such as on a television screen or a monitor. The abilityto create images from virtual camera positions can enable thepresentation of the scene in such a way as to give the operating roomstaff a sense of depth on that display.

This may be done by presenting, in real time, a sequence of images fromvirtual camera positions which sweep from the actual surgeon right eyeposition to the actual surgeon left eye position and back, i.e., thevirtual camera positions are swept over the interocular separationbetween the two eye positions. The sweeping of the virtual camera viewpoint back and forth generates a sequence of images of a scene from aplurality of virtual camera view points. Specifically, at each timestep, an image from a different virtual camera view point is generatedand presented.

The actual right eye position and the actual left eye position, forexample, refer to the left eye position and the right eye position of astereoscopic viewer that is a part of the surgeons control console, inone aspect. The left eye position and right eye position are separatedby the interocular separation.

The display of the images generated by this sweeping of the virtualcamera position over time gives the appearance on the display of smallback and forth head motions, i.e., the displayed images provide a slightrocking back and forth of the scene. This is one of the many depth cuesused by the human visual system and provides a sense of the scene depthto the operating room staff without requiring the use of a stereoscopicdisplay. Additionally, this has the advantage that the imagecommunicates depth independent of the orientation of the operating roomstaff to the display and does not require any special additionalequipment. The virtual camera images may also be created by consideringthe images captured as samples of a light field and using image basedrendering techniques to create the new virtual camera views.

In the aspect just described, the stereoscopic image for the virtualcamera view point was generated using images from stereoscopic endoscope802. In another aspect, a stereoscopic image for the virtual camera viewpoint is generated using a pair of images from a monoscopic endoscope,e.g., an endoscope that has only the right channel of stereoscopicendoscope 802.

In this aspect, 3-D virtualization module 890 performs process 891 to894 for the two captured images to generate a first image for thethree-dimensional image. To generate the second image for thethree-dimensional image, generate images process 896 generates a virtualimage from a view point with the proper ocular separation so that whenthe first image and the virtual image are viewed as a stereoscopic pairof images, the surgeon perceives a three dimensional image.

This approach is sufficient to generate a stereoscopic view of the scenefor a virtual camera view point when there are no sharp edges offore-ground objects in the scene viewed by the monoscopic endoscope.Typically, in a scene of a surgical site, the scene is smooth with nosharp edges and so this process generates useable stereoscopic views. Ifthere is a surgical instrument in the scene, there may be a hole instereoscopic view behind the surgical instrument, but typically thisarea is not of interest to the surgeon. The stereoscopic view with thehole behind the surgical instrument provides the information needed bythe surgeon and so is acceptable.

In the above examples, beam splitter 831R (831L) was configured as abalanced beam splitter. However, in another aspect, the above examplesuse a beam splitter 831R (831L) configured to take advantage of the factthat the illumination of a surgical site is typically accomplished witha light guide, called the illumination channel above, which is part ofthe endoscope. Thus, the illumination is attached to and travels withthe tip of the endoscope. As a result, when the endoscope tip is closeto the tissue surface, the image is much brighter than when the tissueis far away. In this aspect, coated first surface 831R (831L) isconfigured to reflect and transmit different portions of the receivedlight, i.e., the first and second percentages defined above aredifferent.

Beam splitter 831R (831L) reflects M % of the received light and passesN % of the received light, where M % is different from N %. Here, M andN are positive numbers. In one aspect, M % plus N % is equal to aboutone hundred percent. The equality may not be exact due to light lossesand due to tolerances of the various parts of image capture unit 825R(825L). The other aspects of prism assembly 830R (830L) are the same asdescribed above. The processing of the images captured by sensorassembly 820R (820L) is equivalent to that described above for thebalanced beam splitter and so is not repeated.

In the above examples, the image capture sensors were coplanar. However,it was explained that in some aspects, the image captures sensors can bein parallel planes separated by a known distance. FIG. 9 is a schematicillustration of a part of a channel with an image capture unit that hassuch image capture sensors. The configuration of FIG. 9 can be used inany of the aspects and examples described herein and so each of thevarious aspects and examples are not repeated for this configuration.

In FIG. 9, image capture unit 925 includes a lens assembly 901 and asensor assembly 920. Sensor assembly 920 includes a prism assembly 930,a reflective unit 940, and image capture sensors 910, 915. Lens assembly901 includes a plurality of optical elements including an opticalelement that defines an optical stop 970, sometimes referred to as stop970. Light passing through stop 970 is received by a beam splitter 931in prism assembly 930 of image capture unit 925.

A stereoscopic apparatus would include two image capture units asillustrated in FIG. 9. However, as demonstrated above with respect toFIG. 3A, the left and right stereoscopic channels with image captureunits are symmetric, and so only a single channel and image capture unitis described to avoid duplicative description. The other channelincluding the image capture unit is symmetric across a planeintersecting the longitudinal axis of the endoscope with the channelillustrated in FIG. 9.

Beam splitter 931 is positioned to receive light that passes throughstop 970. Beam splitter 931 is configured to direct a first portion ofthe received light to a first image capture sensor 910, and to pass asecond portion of the received light through the beam splitter toreflective unit 940. In this example, beam splitter 931 in prismassembly 930 is implemented as a coated first surface. Thus, beamsplitter 931 is sometimes referred to as coated first surface 931.Coated first surface 931 separates the received light into the twoportions.

Coated first surface 931 reflects the first portion of the receivedlight to a second surface 932 that in turn directs, e.g., reflects, thelight onto first image capture sensor 910. The second portion of thereceived light is passed through coated first surface 931 to reflectiveunit 940. In one aspect, the angle of incidence of the light to coatedsurface 931 is less than forty-five degrees.

Second surface 932 is positioned so that no light other than the lightreflected by coated first surface 931 hits second surface 932. In oneaspect, second surface 932 is a reflective surface that is implementedas one of a coated surface and a total internal reflection surface.

Reflective unit 940 includes a third surface 941 that reflects the lightreceived from beam splitter 931 onto a second image capture sensor 915.In one aspect, third surface 941 is a reflective surface implemented,for example, as one of a coated surface and a total internal reflectionsurface.

First image capture sensor 910 has a top sensor surface in a first plane911. Second image capture sensor 915 has a top sensor surface in secondplane 914. First plane 911 is substantially parallel to second plane 914and is separated from second plane 914 by a known distance d. Herein,about parallel or substantially parallel means that the two planes areplanes within the tolerances associated with manufacturing and mountingthe image captures sensors. The top sensor surface of an image capturesensor is the surface of the image capture sensor that receives lightfrom at least one optical component in the sensor assembly.

In one aspect, a first optical path length from stop 970 to coated firstsurface 931 to second surface 932 to image capture sensor 910 is aboutequal to a second optical path length from stop 970 through coated firstsurface 931 to coated surface 941 to image capture sensor 915. Thus, inthis aspect, image capture sensors 910 and 915 have a common opticalpath length through the front end optical structure in lens assembly 901and about the same optical path length to each image capture sensor insensor assembly 920.

In another aspect, coated surface 941 is positioned so that the firstand second optical path lengths are not equal. In this aspect, imagecapture sensors 910 and 915 have a common optical path length throughthe front end optical structure in lens assembly 901 and differentoptical path lengths to each image capture sensor in sensor assembly920.

In the examples described above, the implementations considered captureand processing of a single set of images in each channel. This was donefor ease of description and is not intended to be limiting. Typically,central controller 260 with camera control units (CCUs) 230R, 230Lcontrols the frequency at which sets of images in each of the channelsarc captured. The processing of the captured images described above isdone in real time, in one aspect, so that the surgeon views a videosequence of stereoscopic images.

Similarly, the different aspects of feature differentiation, enhancedresolution, enhanced dynamic range, variable pixel vectorconfigurations, and extended depth of field were considered separately.However, in view of this disclosure, the different aspects can becombined to achieve the various advantages in combination. The possiblepermutations to achieve the different combinations are not separatelydescribed to avoid repetition.

Also, the above examples, considered an image capture unit mounted at adistal end of an endoscope, either stereoscopic or monoscopic. However,the novel image capture unit configuration can be used in conventionalendoscope cameras as well as in for example, stereoscopic surgicalmicroscopes. Accordingly, the description of implementing the imagecapture unit in an endoscope is illustrative only is not intended to belimiting.

FIG. 10 is a schematic illustration of a distal end of a stereoscopicendoscope 1002 with an image capture unit 1025. Image capture unit 1025includes left lens assembly 1001L, right lens assembly 1001R and asensor assembly 1020 with coplanar right and left image capture sensors1010R and 1010L, in this aspect. In this aspect, stereoscopic endoscope1002, sometimes referred to as endoscope 1002, includes an illuminationchannel 1005. However, any of the various illuminators described hereinincluding internal and external illuminators can be used with endoscope1002. In this aspect, right and left image capture sensors 1010R and1010L are included in a semiconductor substrate 1017, sometime called asemiconductor die or semiconductor chip, which is mounted on a platform1012.

The light from tissue 1003 passes through lens assemblies 1001R and1001L to sensor assembly 1020. The elements in lens assembly 401B (FIG.4B) are an example of lens assembly 1001R and lens assembly 1001L. Thelight that passes through lens assemblies 1001R and 1001L is received bya reflective surface 1041 of a reflective unit 1040. Reflective surface1041 directs, e.g., reflects, the received light onto a top sensorsurface of first image capture sensor 1010R and onto a top sensorsurface of second image capture sensor 1010L. Surface 1041 can be eithera coated surface or a total internal reflection surface.

The optical path lengths to first image capture sensor 1010R and tosecond image capture sensor 1010L within endoscope 1002 aresubstantially the same length. For example, a first optical path lengthfrom lens assembly 1001R to the top sensor surface of first imagecapture sensor 1010R is about equal to a second optical path length fromlens assembly 1001L to the top sensor surface of second image capturesensor 1010L. Thus, each stereoscopic optical channel in endoscope 1002has about the same optical path length to an image sensor.

In this aspect, the surface of image capture sensor 1010R and thesurface of image capture sensor 1010L are coplanar. Alternatively, imagecapture sensor 1010R has a top sensor surface in a first plane and imagecapture sensor 1010L has a top sensor surface in a second plane, wherethe first and second planes are parallel and are separated by a knowndistance (See FIG. 9). In this aspect, the position of the reflectivesurface in the reflective unit is adjusted to compensate for the knowndistance so that the first optical path length to the first imagecapture sensor is about equal to the second optical path length to thesecond image capture sensor. Alternatively or in addition, a lens couldbe used in one or both of the optical paths to create the about equaloptical path lengths.

In the example of FIG. 10 as well as in the alternative arrangement ofthe image capture sensors, the images captured by first image capturesensor 1010R and to second image capture sensor 1010L have the samefocus and depth of field and are spatially registered relative to eachother. Image capture unit 1025 is small and compact and so can be usedin applications that do not have sufficient space to accommodate theimage capture units described above. Also note that this image captureunit is not symmetric about central axis 1090 of endoscope 1002. Incertain applications where there are additional lumens for tools orinstruments, this asymmetry is an advantage. This construction alsoaffords perfect alignment of the imaging areas for the left and righteyes in the case where the image sensor comprises an imaging area foreach eye on a single silicon die or in the case where the device uses alarger imaging device and one does not necessarily use some of thepixels in the region between the two imaging areas.

All examples and illustrative references are non-limiting and should notbe used to limit the claims to specific implementations and embodimentsdescribed herein and their equivalents. The headings are solely forformatting and should not be used to limit the subject matter in anyway, because text under one heading may cross reference or apply to textunder one or more headings. Finally, in view of this disclosure,particular features described in relation to one aspect or embodimentmay be applied to other disclosed aspects or embodiments of theinvention, even though not specifically shown in the drawings ordescribed in the text.

The various modules described herein can be implemented by softwareexecuting on a processor, hardware, firmware, or any combination of thethree. When the modules are implemented as software executing on aprocessor, the software is stored in a memory as computer readableinstructions and the computer readable instructions are executed on theprocessor. All or part of the memory can be in a different physicallocation than a processor so long as the processor can be coupled to thememory. Memory refers to a volatile memory, a non-volatile memory, orany combination of the two.

Also, the functions of the various modules, as described herein, may beperformed by one unit, or divided up among different components, each ofwhich may be implemented in turn by any combination of hardware,software that is executed on a processor, and firmware. When divided upamong different components, the components may be centralized in onelocation or distributed across system 200 for distributed processingpurposes. The execution of the various modules results in methods thatperform the processes described above for the various modules andcontroller 260.

Thus, a processor is coupled to a memory containing instructionsexecuted by the processor. This could be accomplished within a computersystem, or alternatively via a connection to another computer viamoderns and analog lines, or digital interfaces and a digital carrierline.

Herein, a computer program product comprises a computer readable mediumconfigured to store computer readable code needed for any part of or allof the processes described herein, or in which computer readable codefor any part of or all of those processes is stored. Some examples ofcomputer program products are CD-ROM discs, DVD discs, flash memory, ROMcards, floppy discs, magnetic tapes, computer hard drives, servers on anetwork and signals transmitted over a network representing computerreadable program code. A non-transitory tangible computer programproduct comprises a tangible computer readable medium configured tostore computer readable instructions for any part of or all of theprocesses or in which computer readable instructions for any part of orall of the processes is stored. Non-transitory tangible computer programproducts are CD-ROM discs, DVD discs, flash memory, ROM cards, floppydiscs, magnetic tapes, computer hard drives and other physical storagemediums.

In view of this disclosure, instructions used in any part of or all ofthe processes described herein can be implemented in a wide variety ofcomputer system configurations using an operating system and computerprogramming language of interest to the user.

Herein, first and second are used as adjectives to distinguish betweenelements and are not intended to indicate a number of elements. Also,top, bottom, and side are used as adjectives to aid in distinguishingbetween elements as viewed in the drawings, and to help visualizerelative relationships between the elements. For example, top and bottomsurfaces are first and second surfaces that are opposite and removedfrom each other. A side surface is a third surface that extends betweenthe first and second surfaces. Top, bottom, and side are not being usedto define absolute physical positions.

The above description and the accompanying drawings that illustrateaspects and embodiments of the present inventions should not be taken aslimiting—the claims define the protected inventions. Various mechanical,compositional, structural, electrical, and operational changes may bemade without departing from the spirit and scope of this description andthe claims. In some instances, well-known circuits, structures, andtechniques have not been shown or described in detail to avoid obscuringthe invention.

Further, this description's terminology is not intended to limit theinvention. For example, spatially relative terms—such as “beneath”,“below”, “lower”, “above”, “upper”, “proximal”, “distal”, and thelike—may be used to describe one element's or feature's relationship toanother element or feature as illustrated in the figures. Thesespatially relative terms are intended to encompass different positions(i.e., locations) and orientations (i.e., rotational placements) of thedevice in use or operation in addition to the position and orientationshown in the figures. For example, if the device in the figures isturned over, elements described as “below” or “beneath” other elementsor features would then be “above” or “over” the other elements orfeatures. Thus, the exemplary term “below” can encompass both positionsand orientations of above and below. The device may be otherwiseoriented (rotated 90 degrees or at other orientations) and the spatiallyrelative descriptors used herein interpreted accordingly. Likewise,descriptions of movement along and around various axes include variousspecial device positions and orientations.

The singular forms “a”, “an”, and “the” are intended to include theplural forms as well, unless the context indicates otherwise. The terms“comprises”, “comprising”, “includes”, and the like specify the presenceof stated features, steps, operations, elements, and/or components butdo not preclude the presence or addition of one or more other features,steps, operations, elements, components, and/or groups. Componentsdescribed as coupled may be electrically or mechanically directlycoupled, or they may be indirectly coupled via one or more intermediatecomponents.

1-11. (canceled)
 12. An apparatus comprising: a first image capturesensor comprising a first image capture sensor surface; a second imagecapture sensor comprising a second image capture sensor surface; and anassembly positioned to receive light, the assembly comprising: a distalface through which the received light enters the assembly; a beamsplitter configured to separate the received light into a first portionof light and a second portion of light; a surface of the beam splitterconfigured to reflect the first portion of light onto the first imagecapture sensor; and a reflective unit positioned to receive, directlyfrom the beam splitter, the second portion of light and to reflect thesecond portion of light onto the second image capture sensor, wherein afirst optical path length from the distal face to the first imagecapture sensor surface is about equal to a second optical path lengthfrom the distal face to the second image capture sensor surface.
 13. Theapparatus of claim 12, further comprising: a controller coupled to thefirst and second image capture sensors; the controller being configuredto generate an enhanced image from a first image generated by the firstimage capture sensor from the first portion of light and a second imagegenerated by the second image capture sensor from the second portion oflight.
 14. The apparatus of claim 13, wherein the enhanced image hasreduced noise and greater apparent resolution relative to either thefirst image or the second image.
 15. The apparatus of claim 13, whereinthe enhanced image comprises a plurality of pixels, and each pixel ofthe plurality of pixels is based on a pixel from the first image and acorresponding pixel from the second image.
 16. The apparatus of claim13, wherein the first image and the second image are the same.
 17. Theapparatus of claim 12, further comprising: a controller coupled to thefirst and second image capture sensors; the controller being configuredto generate an interpolated image from a first image generated by thefirst image capture sensor from the first portion of light and a secondimage generated by the second image capture sensor from the secondportion of light.
 18. The apparatus of claim 17, wherein theinterpolated image has reduced noise and greater special resolutionrelative to either the first image or the second image.
 19. Theapparatus of claim 17, wherein the first image is offset from the secondimage by half a pixel.
 20. The apparatus of claim 17, wherein the firstimage and the second image are different.
 21. The apparatus of claim 12,wherein the first and second image capture sensors comprise differentareas of an image capture sensor chip.
 22. The apparatus of claim 12,wherein the first and second image capture sensors are part of astereoscopic endoscope.
 23. The apparatus of claim 12, wherein the firstportion of light and the second portion of light are equal.
 24. Theapparatus of claim 1, wherein the beam splitter is a balanced beamsplitter.
 25. A method comprising: receiving light through a distal faceof an assembly; separating the received light into a first portion oflight and a second portion of light by a beam splitter; reflecting thefirst portion of light onto a first image capture sensor by a surface ofthe beam splitter; and reflecting the the second portion of light onto asecond image capture sensor by a reflective unit; wherein a firstoptical path length from the distal face to the first image capturesensor is about equal to a second optical path length from the distalface to the second image capture sensor.
 26. The method of claim 25,further comprising: generating, by a controller, an enhanced image froma first image generated by the first image capture sensor from the firstportion of light and a second image generated by the second imagecapture sensor from the second portion of light.
 27. The method of claim26, wherein the enhanced image has reduced noise and greater apparentresolution relative to either the first image or the second image. 28.The method of claim 26, wherein the enhanced image comprises a pluralityof pixels, and each pixel of the plurality of pixels is based on a pixelfrom the first image and a corresponding pixel from the second image.29. The method of claim 25, further comprising generating, by acontroller, an interpolated image from a first image generated by thefirst image capture sensor from the first portion of light and a secondimage generated by the second image capture sensor from the secondportion of light.
 30. The method of claim 29, wherein the interpolatedimage has reduced noise and greater special resolution relative toeither the first image or the second image.
 31. The method of claim 29,wherein the first image is offset from the second image by half a pixel.